If you inherit a house with a leaky roof, it isn’t your fault the roof is leaking. It is your responsibility to fix the roof now though.
It isn’t just about the rich people who profited off slavery. It is also about the net effect of the entire society’s racism. Things like black children being given worse education or black families being prevented developing generational wealth through home ownership. We didn’t cause that, but we all inherited the house with the leaky roof and it’s our responsibility to fix it.