Figuring out if two posts are identical is going to be a challenge
I think I could deal with duplicates pretty easily by checking if different properties of the post are identical, the more are the less likely it is that the post will be shown.
I think a bigger problem is the comments. What if a user reposts a post from r/astronomy on lemmy.world to c/astronomy on lemmy.ml? Which one do we show, and what do we do with the comments of the other? Merge them together, or just leave them? Maybe show a button on the post when there’s a duplicate und the user can switch instances?
Lots of figuring out to do, but it sounds fun!
Edit: The Group-by-Hashtag thing is also a good idea! That would of course make the whole thing less of a headache, but to make that work a lot of users need to do that.