Probably something accessing your microphone and making the headset switch to HSP/HFP. Correcting the microphone input usually allows me to switch back to A2DP, but not always unfortunately. I’ve had it happen a few times at random too (some games, a chat program). I still don’t know what causes my issues exactly, but after a few updates and editing my Pipewire config it doesn’t seem to happen as much anymore.
I dug in deep and I think completely removing the HSP and HFP roles from the Wireplumber config file solved the issue. I don’t know what to do if you’re on Pulseaudio, and there’s no GUI setting to do this, but it may help you on your quest to figure this stuff out.