It’s not necessarily true that the baseband has unfettered access to the entire phone. Pixel devices for example use a special IOMMU to restrict what the baseband can access, forcing it to go through a specialized interface only. It still requires more work for a compromise of baseband to get control of Android.
First you need to exploit the baseband. Then, you need to exploit the kernel.
Now, that’s a significant attack surface, but the point stands that many phones now have some compartmentalization because of this risk. This has been a concern for some time and newer designs are trying to mitigate it.
Here’s a security evaluation of the pixel which shows that a compromise of the modem does not equate to an immediate compromise of the device. The modem must be restricted in what it can access of the application processor.