You also can’t prevent processes from manipulating each others inputs/outputs.
That’s one of those pseudo-problems. In theory, yeah, a bit more control over what apps can and can’t access would be nice. In reality, it doesn’t really matter, since any malicious app can do more than enough damage even without having access to the Xserver. The solution is to not run malicious code or use WASM if you want real isolation. Xnest, Xephyr and X11 protocol proxy have also been around for a while, X11 doesn’t prevent you from doing isolation.
Trying to patch sandboxing into Linux after the fact is not only not giving you isolation that is actually meaningful, it also restricts user freedom enormously. Screenshots, screen recording, screen sharing, keyboard macros, automation, etc. All very important things, suddenly become a whole lot more difficult if everything is isolated. You lose a ton of functionality without gaining any. Almost 15 years later and Wayland is still playing catch up to feature that used to “just work” in X11.