You didn’t list your distro, but I had the same issue on Pop!_OS. I fixed it by modifying the ALSA properties in /usr/share/wireplumber/main.lua.d/50-alsa-config.lua. I’d first try uncommenting headroom and changing it to 1024, then logging out and back in. If that doesn’t work, go 2048. If that still doesn’t work, then I’d maybe try adjusting period-size.
For me, a value in the ~1200 range for headroom fixed it completely. But it will entirely depend on your sound card/hardware. And note, these values will probably be overwritten with any wireplummer updates.
Also keep in mind, increasing headroom will increase audio latency, so try not to increase it too drastically.