An X11 application connects to Xwayland just like it would connect to any X server. Xwayland processes all the X11 requests. On the other end, Xwayland is a Wayland client that connects to the Wayland compositor.
The X11 window manager (XWM) is an integral part of the Wayland compositor. XWM uses the usual X11 window management protocol to manage all X11 windows in Xwayland. Most importantly, XWM acts as a bridge between Xwayland window state and the Wayland compositor’s window manager (WWM). This way WWM can manage all windows, both native Wayland and X11 (Xwayland) windows. This is very important for a coherent user experience.
There is also rootless mode.
In rootless mode, each foreign window is a first-class resident among the native windows. Foreign windows are not confined inside a native window but act as if they were native windows. The advantage is that one can freely stack and mix native and foreign windows, which is not possible in rootful mode. The disadvantage is that this mode is harder to implement and fundamental differences in window systems may prevent some things from working. With rootless Xwayland, the Wayland compositor must take the role as the X11 window manager, and one cannot use any other X11 window manager in its place.
That’s the toolbox we have right now. To make existing X11 window manager work exactly as before on Wayland would be another level. And why we? It would make Wayland way more complex than it is and if we want to use old software anyway, why do we develop Wayland at all? (The “WE” is not me, i’m speaking for devs.)
Some existing X11 window manager are already working on XWayland (think of Qtile) or are ported over right now. The simple ones could be rewritten from scratch, even if its not exactly the same (think of Sway). There will be some left. But that’s okay. Window manager itself are not hugely complex and different from each other, only a few logic and the language differs greatly. We have already lot of variety. It’s not like we would leave KDE or Gnome behind.
My opinion: We don’t need to make Wayland even more complicated (for Wayland devs and for compositor and app devs), just for a few legacy tools.