Personally, I love visiting France, but won’t be visiting this summer due to the overcrowding that comes with the Olympics. I’m actually sort of yearning to revisit Paris but I guess I’ll have to wait until next year or perhaps winter. So I’m one of those visitors who’s (temporarily) dissuaded from going due to the event.
I will say that I think looking at it as an economic “thing” is fundamentally wrong. Hosting the olympics should be about offering the locals a chance to experience world class sports events and the civic pride that comes with putting your country on the map. If you only want to do it for economic benefits, then you really shouldn’t host it.