I had been self-hosting stuff on my QNAP NAS for years before it died due to the infamous Intel clock drift issue and now I am in the process of making a DIY NAS (last few parts are coming in this weekend). I don’t have answers to all your questions but I’ll try my best with the experience that I have.
It is absolutely possible to mix your usecases on one machine, with the caveat that if you’re running on less-powerful hardware (like a off-the-shelf NAS), some of your services might be competing with each other for resources. CPU usage and disk access times (especially with a RAID 5 HDD array) can all impact performance. My QNAP NAS did start to bog down a few times with both Jellyfin and Nextcloud running at full tilt, but it was generally pretty usable.
Most NAS products support docker images so I wouldn’t worry too much about NAS vs PC in this case. Also, docker-compose is your friend. Write your yaml file once and it will make for easy setup and upgrading.
Dude, I am with you on dead-end products. The death of my QNAP NAS has caused me lots of headache and I basically swore off products that I can’t upgrade and fix myself. The problem is price. The cheapest x86 PC that I personally think will handle multiple usecases (media server, Nextcloud, SAMBA, maybe a Valheim server or a VM when I need it) costs roughly around $650-$750 depending on your build. You can probably find a Synology or QNAP NAS for about $500-$550. Granted, they most likely aren’t going to be anywhere near as powerful as a DIY x86 PC, so I think its worth going the DIY route. Those prices do NOT include the drives either, so be sure to factor that into your calculation. If you’re curious, here’s one of the cheaper builds I was considering building: pcpartpicker.com/list/rtqDbK. Ultimately I decided to go for a crazier build because I did not want slow HDDs anymore: pcpartpicker.com/list/Lm92Kp
You mean running a media server on your laptop, but pointing the media libraries to a Samba share on a NAS? I did that for years with my QNAP NAS and a little Intel NUC running Plex. The only issue is that you won’t get incremental media library updates whenever you add new files into the Samba folder. Usually, Plex (and Jellyfin) can detect file changes if the media library is local and automatically process only those files instead of rescanning the entire media library. Over Samba, there’s no such automatic detection so whenever you add a file, you have to manually trigger a full rescan in order for it to pop up in your media library.
I believe Unraid does this. I have not tried it myself and I plan on going with ZFS for my DIY NAS.
I don’t have any resource recommendations, but personally I’ve taken the docker-compose approach which helps quite a bit for isolation. For media servers, you only need to give read-only access to the volumes hosting your media storage. It is also recommended to put media servers like Jellyfin behind a reverse Nginx proxy because Nginx has been battle-tested in terms of security and Jellyfin’s web server has not. You can use docker-compose to easily spin up a Nginx proxy alongside your media server and have them contained in their own isolated network.
Do not open any more ports than is necessary to host your services. This means even remote administration should not be available via your public IP. Learn how to setup Wireguard so that if you’re away from home, you can quickly VPN into your network and do remote administration. If you’re using SSH, make sure you disable password authentication and only rely on SSH keys. I am sure other people can add more, this is just the basics.