Last time I had a vm with physical output was years ago, and it was an involved process. The thing you are looking for is PCI pass-through. Unless you are thinking closer to just running something like virtual box full screen?
Did you need your server to have hdmi output instead of running a normal jellyfin client? Would you consider a desktop os running server containers in docker or podman?