It depends on what your level of confidence and paranoia is. Things on the Internet get scanned constantly, I actually get routine reports from one of them that I noticed in the logs and hit them up via an associated website. Just take it as an expected that someone out there is going to try and see if admin/password gets into some login screen if it’s facing the web.
For the most part, so long as you keep things updated and use reputable and maintained software for your system the larger risk is going to come from someone clicking a link in the wrong email than from someone haxxoring in from the public internet.