“To taste” isn’t meant literally in the phrase. It’s an idiom, and you can use it in another context (e.g. add accessory to outfit to taste) as a clever way of saying “to one’s preferences.”
Although technically it’s original use in recipes isn’t actually meant that way, and actually means to keep adding and tasting the food as you go until it feels sufficient.
I also got tripped up on this when I was first learning English so don’t worry too much OP! It’s a fickle language.