Usually, difficulty to learn a language is caused by:
not speaking a similar language. Because then there’s less features to correctly transfer from your other languages to the target language.
relative lack of resources available to learn said language. Including native speakers willing to chat with you in the language, instead of shifting to a common language.
how much time and effort you spend productively interacting with that language, and how necessary it is for you to learn it.
Another thing, relevant in the light of other comments: language is mostly what’s inside our heads, not our mouths. Small differences in the vocal tract can affect a bit our pronunciation - like the pitch, or ability to pronounce specific sounds, but in the big picture they’re mostly irrelevant and “abstracted out” - it’s like when you’re writing, it doesn’t stop being written [Mandarin|English|Spanish|etc.] because you used a red pen instead of a black pen, you know?