A lot of it will depend on your output device; cheap headphones will wreck audio quality.
I remember the bad old days when .mp3 files for streaming were often 128kbps (or less!); I could absolutely hear audio artifacts on those, and it got significantly worse with lower bitrates. 320kbps though seems to be both fairly small, and I can’t personally tell the difference between that and any lossless formats.