The problem is libertarian groups themselves don’t stand up against anarchists joining them. I remember Gary Johnson getting booed at an official libertarian party gathering for saying he believes in driver’s licences.
Also we already know how libertarianism ends, with robber barons controlling everything and people living in company towns. It’s a terrible political ideal.