I understand what you're saying but disagree with your ultimate conclusion.
The average modern-day American did not directly benefit from slavery and many didn't even come to the country until after slavery had been long since abolished. Some of those people were also treated as ethnic minorities though they may be seen as "white" now. Which is to say, they were terribly disadvantaged in the American economy because they were immigrants or descended from the "wrong" background.
Any reparations that are to be made should be made by the perpetrators or those who have directly benefited from said perpetrators' actions. To tax other impoverished lineages in order to provide reparations to another group which had it even worse is unjust.
I do firmly believe that the country needs to address its systemic issues, but I don't believe that taxing the majority to give cash to the aggrieved is the solution.