I’ve been on the hiring end of those conversations before, and frankly I prefer it when a candidate withdraws. It saves me the time and effort of an interview and let’s me focus on other candidates.
Don’t forget, it’s an inter-view-- you’re vetting them as a potential employer just as much as they’re vetting you as a potential hire. It’s completely reasonable to tell them that after further consideration, you don’t think the you’re a good fit for the job and that you’d like to withdraw your application, thanking them for their time and consideration. It’s more professional to be respectful of everyone’s time and withdraw since you already know you don’t want the job.
If they booked you plane tickets or something to fly you in for a face to face…? Eh, they might have a beef with that. You’d have wanted to withdraw before it got that far.