I can’t help with a self-hosted captcha, but I do know that hCAPTCHA claims to be more privacy respecting than reCAPTCHA. They also have a 1:1 comparability layer with the reCAPTCHA API so it should be a drop-in replacement without too much effort.
I’m interested to hear if anyone chimes in with a self-hosted solution, but I’d imagine a managed solution would probably be best for an application of any size if you’re worried about bots.
Also, while I agree with the other poster that bots may be better than humans at solving captchas, I do want to say that they’re better than nothing. Just like I wouldn’t leave my front door unlocked (even though house doors are easily picked / broken), a simple deterrent is better than nothing. A site I was working on went from hourly spam to none at all with just a simple Cloudflare captcha.