Prediction is a hard problem when coupled with caches. It relatively easy to say that no speculative instruction has any effect until it’s confirmed taken if you ignore caches. However caches need to fetch information from memory to allow an instruction to evaluate, and rewinding a cache to it’s previous state on a mispredict is almost impossible. Especially when you consider that the amount of time you’re executing non-speculative code on a modern processor is very low.
Not having predictions is consigning yourself to 1990s performance, with faster clocks.