If you’re on a budget, check out X99 socket Xeons. You can pick up Mobos and chips for super cheap. 10+ core hyper-threaded Xeons with solid clocks and a motherboard for 120-180 bucks total. Support 64 GB of RAM, more if you have a proper server board.
For transcoding, depending on the codec, dedicated GPU is best.
I’m not sure about Plex, but I know on Jellyfin, the new Intel Arc GPUs are really great for encoding, not too expensive for the lower end cards either, and low profile options for smaller rack cases.