Jealousy is definitely the biggest sticking point for me. Every woman I’ve been with has always had a few other dudes sniffing around at any given time waiting for their chance. I on the other hand have gone years between relationships because I have a hard time meeting women that I’m interested in who also want to be with me. If I could easily find someone else to hook up with while my gf was out doing it I wouldn’t really have that big of a problem but I would have a huge issue if I was stuck at home by myself while she was out potentially finding a replacement for me.
Also I think you have a good point that if we didn’t have to invest so many resources into a relationship more people would probably be more okay with it.