Exactly. They’d have to prove you were specifically going there to get an abortion.
Cops can’t stop you because you were on your way to a bank, just because they feel you might want to rob it. You have to have actually done something illegal in the first place.