Based on anecdotal evidence, “I refuse” may be more common than you think. I live in Switzerland, and out of all the expats I know who have been here a long time (20+ years), a large percentage (over half) of them still can’t speak German. At all. Like, they can maybe say “thank you” but that’s about the extent of their vocabulary, and many of them actually seem to be quite proud of the fact that they’ve made absolutely zero attempt to integrate into the local community or culture.
Now, although I’m fully aware that there are a hell of a lot of differences between Switzerland and the US, it still wouldn’t be at all surprising to me if there were large groups of immigrants in the US who similarly resist learning English.