If your headphones are crappy, a better source won’t really help. It’s not an or here, headphones make most of the difference. Only decent headphones will appreciably benefit from a good source.
MP3 isn’t ideal but, unless you have rather good headphones, 320kbit/s MP3 is pretty damn near transparent.