Yeah… don’t pay any attention to this kind of nonsense. Cats are obligated to carnivores. They don’t have the machinery to process a lot of non-meat foods. If they occasionally get into some things, it shouldn’t cause a problem.
Ultimately, just listen to your vet.
So far as the milk and cats thing… I think the issue is that they love it, they’ll almost always go for it, but many if not most are lactose intolerant… so too much will cause issues. If you had a cat who could process it, then great!