It’s your private key, but yes, you would need to keep it secret just like you would an SSH key.
The benefits of a VPN are that you don’t need to open ports up to the internet and rely on your individual services to be secure. Your VPN would authenticate users and ensure that the communication over the tunnel is encrypted (useful if you don’t want to set up SSL/https). They can also hide what services you are hosting or even hide the fact that you are even running a VPN.
Private keys are going to be far more secure than passwords since you really can’t brute force them in the same way you can passwords. Getting ahold of someone’s private key is probably going to be far more difficult than guessing their password. Even if an attacker were to get ahold of your private key, they would still need to contend with the security of your service, e.g. logging into it, which would be no worse than not having a VPN.