You're allowed to like a story you grew up with as a child and also dislike its bigot author, they're not mutually exclusive. Talking about Harry Potter doesn't give Rowling magical transphobe powers; Voldemort logic doesn't work in real life. The rightsholders have already taken great strides to distance the HP property from Rowling and adopt it to be more inclusive in spite of her TERF bullshit. It's not a hate crime to like a story about child wizards anymore.
If people want to geek out about some books or movies they like, they should be allowed to do so without the insinuation that they're by default enabling transphobia or something. But the beauty of the Fediverse is that your community has just as much right to exist as any other, so as long as you can maintain a healthy, hate-free community that isn't posting a bunch of pro-Rowling bullshit, I say go for it. Anyone who would block your instance for merely existing probably isn't worth your time, anyway.