I'm a firm believer that a not insignificant portion of people had one or two really shit math teachers at some point, decided that they're bad at math because of it, and then proceeded to just give up. Very often it was specifically related to fractions.
The math professors at my uni were fantastic, and I saw many friends who always thought they were bad at math have lightbulb moments where something finally clicks.