Do you mean it constantly does it when a monitor is turned off or that when you initially turn off a monitor, it rearranges all windows to fit on the remaining monitor.
If the first, I’m not sure what the problem might be, but the second is pretty normal, I think. The card sees that the display was detached and moves your windows to the attached display so you can see them.