At least on the Fairphone you can trivially swap out the USB-C port when it breaks. And using it for audio moves that from an if to a when, in particular on a phone designed to be used for at least 5 years.
But that’s okay, since you can swap it out. On any other phone: Fuck USB-C audio! The port is fragile enough as it is.