In my state, the county Board of Elections (and their staff) is in charge of the voter rolls. They are employed by the state to do that job, and can be fired if they do a bad job. They also can have access to state vital records, and can take deceased voters off the rolls right away.
On the other hand, if we leave this job to unelected busybodies, then there is nothing keeping them from adding some extra names to their list, to harass neighbors they don’t think belong there.