No, they don’t annihilate. The electron will scatter off the other particle, though any differences in charge will of course affect the scattering. For example, an electron and a proton could become bound to make a hydrogen atom, but this couldn’t happen with an anti-proton. Any nuclear reactions (specifically electron capture) would be affected too.
In the case of free anti-neutrons, there’s a chance the anti-neutron could decay into an anti-proton and a positron. If this were to happen during the collision with an electron, the electron could potentially annihilate with the positron.