Squirrels can clean out a feeder pretty quickly. Not as fast as deer can, but much faster than the birds.
So it’s a pain in the ass to go fill it back up, and it costs money. A person gets a bird feeder because they want to watch birds. There are cheaper ways to feed squirrels, if you like squirrels.
Both squirrels and birds can build nests in your home. Squirrels can chew their way into your attic, then you risk them chewing through wires. Birds nest in your dryer vent or bathroom vent. A nest in the dryer vent is a fire hazard. And they can introduce bird mites into your home. It’s like having a bed bug infestation except you can’t see them, their bites are hella itchy, and at least they can be dealt with by multiple rounds of thorough vacuuming. Ask me how I know.
I used to love to keep a bird feeder and watch the bird party on a snowy day. But I wasn’t out to feed the deer, and the mite problem erased any lingering feelings about feeding birds.