Here’s my advice. The most important things are that you have a free GPU slot and another PCIE slot on whatever you get. You’re going to want a GPU for transcoding when disk space gets tight. You want the extra PCIE slot for a sas card. Disk shelves are surprisingly cheap, and you can keep adding disk that way. They daisy chain.
As far as the OS, I’m partial to Unraid and Truenas but seriously, anything you are comfortable with will work.