It’s not just being attracted, it’s being attracted conditional on the person having x characteristic (that they might not even want to have). That’s the very definition of fetishizing/objectifying and it’s not toxic to criticize it.
I will grant that this is more of an issue for relationships than for hookups etc. As a wise woman once told me- chasers are gross, but they also help pay her rent, so really it’s impossible to say if they’re bad or not.