There are some places that have a crazy office politics environment (2007 Microsoft, I’m looking at you). If that is the case, I would just change jobs.
However, if you just have annoying coworkers who are off topic, I would just steer the conversation back to work items (“oh, before I forget, can you tell me about [work related item]?”), then leave the conversation. Other tactics that work for me: look really awkward during personal conversation, never ask how someone is doing, wear headphones.
Someone once told me it’s important to care, but not that much. This has also helped me.