It could just ask before removing shit. Remove the permissions, freeze the app, prompt the user to confirm they meant to install it from somewhere other than the playstore. Hell, since it can detect F-Droid is installed, maybe use some context clues and ask the user to confirm this app was installed from there?
More importantly, can you tell it to ignore certain apps? I don’t know, I’ve had Play Protect turned off forever. If not, that’s absolutely asshole design.