1- They’re into your smell and trying to rub themselves on something that smells like you,
2- they’re trying to get your attention,
3- if the litter is also in the bathroom, they could be cleaning their paws on the towel. I’ve seen cats scratch everything in the room after using the litter.
From the way you phrased the question my money is on #2, but you know better. It might be time to get a higher towel bar or close the bathroom door