Yeah, which is why I think storing remote user and instance public keys might be better. Then that can be used to authenticate the migration request (it'd probably need to be an extension to the activitypub standard).
The biggest problem I see is that an instance doesn't know about all the instances that have data pointing to them. So how does it communicate the changes to everyone? The mastadon way is probably the sensible way to do it, despite not supporting the loss of control of domain scenario.