That just comes with the territory of being wealthy, which he lucked into thanks to Peter Thiel taking a liking to him (probably because he wanted to fuck him). People confused the companies he invested in which happened to be successful despite him, and would probably more successful if he weren’t involved- see SpaceX when Shotwell took over day-to-day operations.
Lots of people, otherwise smart people, smell someone with money and say, “I want to go to there” because they think wealth can be transmitted through close contact.