I ran JF for about a year on a Pi 2B. Transcoding off at the server. No issues at all playing any file using Direct Play - including large 4K rips. I moved to an Odroid C2, again, absolutely no issues with playback.
If you’re seeing trouble with Direct Play I’d bank on it being network or storage related rather than the power of the Pi. E.g. the network hasn’t got enough throughput to serve the files. In Direct Play you need very little in terms of server resources as it’s handed off to the client.