Other then the slowly increasing log file (if you use fail2ban for example), it will take thousands of years to actually hack you through this method as long as root auth is disabled and authentication is only via SSH keys, I wouldn’t worry about it.
It is possible to tighten the security of a machine to the point it is no longer usable. It is important to secure our devices but we cannot forget about convenience, so the trick is to tighten it but also make it so you don’t have to jump through a number of hoops till you get to your destination.
I for example, wouldn’t use your method because it would make it difficult to use some services I host from my phone.
Port knockers for the most part aren’t worrying. In an ideal situation, the only ports that should be open are 22, 80, 443 and using a reverse proxy to mask headers. (Poor configuration for example, go to Shodan and type bitwarden in the search bar and see how many people expose their instances to the world carelessly without an SSL cert) and the occasional UDP for game servers/media servers.