Can’t answer NixOS bits as I’ve not used it, but a few points to shoot off:
Starting with a Mastodon instance should not be the first step. Setting up an nginx webserver that says hello world! should.
There’s not much to learn about using SSH as much as setting it up. Key-only auth is a must. Changing the port to something high is a convenience - I hate my logs getting spammed by all the bots. Some are more than annoying enough. I have also enabled rate limiting on my SSH port for that reason alone, using iptables. I really need to have a look at nftables as that is faster and better.
Domain is only really necessary if you want to reach your stuff externally. Mastodon would count as such usecase. Whether you go proper paid or free - up to you, but do keep in mind the recent Mali situation into account. Domains are cheap anyway.
As a sidenote - you can set up your own authoritative DNS, your own VPN and then use whatever domain you desire. You can make your internal DNS serve your internal website on google.com if you wish :) It would, of course, only work on devices using your own DNS as the resolver. You would also most likely want to roll your own CA for that, but I’m getting distracted here :)
In short - go for it! The worst that can happen is you will learn invaluable skills and have to start over.