The ai is trained on recordings of his voice which they have not secured the rights to though.
Not really the case with the latest models, a couple of seconds of audio are enough to clone a voice, as you can essentially remix it from the all the other training data, you don’t need that persons specific voice for training anymore. This is more a personal rights and trademark issue than a copyright one.