I don’t think it’s wasteful to have both KDE and Gnome. It’s healthy competition and as you say, innovation.
However the job of a distribution is to gather upstream software into a meaningful OS, and rewriting everything that should be an upstream software shared with other distributions is a distraction.
So Unity was unnecessary “not invented here” syndrome. Just like Snap is.