I am just starting so take this not as a recommendation but as an option. I am familiar with Linux but do not work in IT.
I got myself a used desktop as a starting point. It can handle 2x 3.5” drives, one 2.5”, plus an NVMe. You could buy an adaptor and change the DVD drive for another 2.5” caddy, but more on that later. It came with 8GB of RAM, but it can handle 64. I spent something like $250 including cables, bolts, caddies, but not drives.
If you watched the video, you’ll notice the CPU has video transcoding acceleration and encryption acceleration too. It comes out ahead of modern N100 CPUs being widely used for home NAS these days, and draws a minuscule amount of power while idle. Indeed, most of the idle power draw for my machine comes from the drives.
So pros:
can host a decent amount of services
upgradable (PCIe slots and up to 4* spindles)
the fourth needs you to convert the DVD to a caddy, but then you need to get an expansion card to add another SATA port, but will allow you to go RAID 10 or z2/6.
small and mostly silent, low power draw
2x M.2 slots, one for NVMe and the other for an expansion card (like a Coral TPU or Wi-Fi)
cheap
Cons:
3x onboard SATA ports, 3x drive bays means you’re rather stuck with RAID z1/5
lower reliability from a used unit
1x 1 Gb Ethernet port onboard only
For software, I’m using TrueNAS scale. It’s easy to install and configure, there’s good documentation and a support forum, can run docker containers and VMs. Lots of administration quality of life tools built in that you don’t need to build. Plus it’s Linux and I can tinker with it if the need arises.
To get to what you want, you could install an M.2 A+E to SATA adaptor and a slim DVD to 2.5” caddy to come up to 4 drives, add memory, a multiport multigigabit NIC, an NVMe and 4 drives and you’d be set. VMs for your firewall, VPN, pihole, dockers for the rest.