There are definitely people who could upgrade from a Model 3 to a Model S, and they are choosing a different car because of Musk.
Furthermore, a lot of people have decided that supporting established car companies, with all their faults, is still better than supporting Musk. And they may choose to switch to the Bolt, Leaf, Kona, Ioniq or similar.