It’s true that a lot of peoples (maybe most?) today live in a place which they took by force from someone else, though you don’t have to look far to find areas that are still inhabited by the first people that arrived there. Still, for a fair comparison you need to separate between those that took areas by force either from necessity (e.g. they were displaced themselves) or otherwise before any kind of international regulation existed.
You cannot compare a tribe or small kingdom taking land by force 2000 years ago to a modern state annexing land, just like you cannot compare the sacking of a city 1000 years ago to a modern genocide. The world has changed.