Certs only prevent others from making it look like it was you, they don’t stop someone from exploiting a vulnerable webapp you might be hosting, or using a misconfigured mail server as a relay.
If you have anything open to the public, then you either have to keep it read only, or stay on it to make sure it’s updated, secured, sanitized, and so on.
Personally, I’ve switched to using client side certificates, so everything is effectively “not public”.