Some people are going to be offended unless you participate in their games. In the army I had someone tell me he was going to punch me if I didn’t drink at a get together. So I drank. I probably would’ve anyway, but he was super weird about it.
Redirect conversation, be noncommittal, let yourself get pulled away by duties, etc. Eventually if you’re lucky you just won’t be in the clique any more. But they’re never going to understand what you’re telling them without offense because they just don’t want to. There is no polite, easy way to do this that isn’t going to upset them. They will feel like you think you’re better than them.