That is the normal case in Germany. Our small town has about 10 districts and they all use the school, but spread over different rooms in different parts of the building. Your invitation contains your district number and there’s ample pointer where to find which district. And in case you lost your invitation and have no idea about your district, we can always look you up in the voter list, which contains all adult residents.
I take it your districts are bigger than several hundred people? In bigger cities we use several buildings, but any voting place should still be able to direct you to the right district. In case the local volunteers are unsure, they can always call the voting office, which contains the guys that planned everything and they clear things up, but that’s rarely necessary.