You’re actively having people ask who their friends are, and they’re being guarded about it.
If you felt like you might be kicked out of a group of friends that you really liked, because of your association with someone THEY didn’t like; wouldn’t you be a little guarded too?
It doesn’t sound like you’re friends with this person at all. So if not, just stop pretending they’re your friend. Because if being friends with you is dependent on them NOT being friends with someone else – then you aren’t a friend.