I wonder if these land-holders live in an HOA. And if they do, I'm sure they're in violation of something. Racking up fines they aren't told about. Their HOA contract gets sold to a corporation to manage it. The corporation sees the ludicrous fines and repossesses their house as in the new HOA contract. Evict the land-holders.
It's not realistic because rich people don't have consequences but it'd be a nice sense of justice.