Bash and a dedicated user should work with very little effort. Basically, create a user on your VM (maybe called git), set up passwordless (and keyless) ssh for this user but force the command to be the git-shell. Next a simple bash script which iterates directories in this user’s home directory and runs git fetch —all. Set cron to run this script periodically (every hour?). To add a new repository, just ssh as your regular user and su to the git user, then clone the new repository into the home directory. To change the upstream, do the same but simply update the remote.
This could probably be packaged as a dockerfile pretty easily, if you don’t mind either needing to specify the port, or losing the machine’s port 22.
EDIT: I found this after posting, might be the easiest way to serve the repositories, in combination with the update script. There’s a bunch more info in the Git Book too, the next section covers setting up HTTP…