The racists are going to be the vast minority of Ye’s audience. Statistically negligible. Some people are just surface-level low-intelligence beatheads who like the music.
R. Kelly had to go to jail to get people to stop listening to his music, and I don’t recall people advocating for pedophilia on his behalf.