You might take a hit to your reputation with that company; but if they already were presenting enough red flags for you to back out of an interview; I wouldn’t consider that to be a problem.
If you cancel an interview it’s not a big deal. No money changed hands. No agreement was ever reached. Their emotional feelings are irrelevant; the whole point was to help both sides decide if you could work with the other.