Nobody would give two shits if that was the problem. The problem is that their behavior endangers the safety of others and the integrity of society.
I don’t just mean people who cannot be vaccinated. Without vaccines, lockdowns like the ones during the covid pandemic would be in effect pretty much all the time. We have functioning vaccines against diseases that are much worse than covid. We got lucky in 2020.