Sometimes it can’t connect to the server (which is a completely stupid necessity).
That’s where it does the voice processing. The only processing it does on-device is the wake word and taking commands. Actually figuring out what you mean is done in The Cloud. Doing that on-device would not only make the devices significantly more expensive, but they would also rapidly become outdated.
The rest of your complaints are valid and I’ve experienced them all myself to boot.