Some oppressors (indirectly, but essentially) started drawing lines one day and agreed that they would each get to farm the humans in their own territory.
Control goes back further than just territories to tribes. The tribe identity is only later tied to specific locations. Tribes formed because pooling resources burdens and learning was more efficient than doing it all yourself. From there, the tribes expanded and joined together and eventually settled into one location. So I disagree that oppressors just decided one day.