Antisemitism is a misnomer, yes. It is referring specifically to the hatred of Jews, not semitic people in general.
The term originated around 1879, when it got popularized by a group of fantics around German journalist Wilhelm Marr by founding the “Antisemiten-Liga” (league of antisemitics). And since these guys were not the brightest, they got that detail wrong.
The term was subsequently used by Nazi Germany and they didn’t correct the inital misnomer either, so now we are stuck with that, I guess.