I generate a unique key pair (or token) for each service that I want to access from the host machine. I see no issue with storing that dedicated private key locally in plaintext (obviously in a folder where only the required user can read it and I except it from backup and versioning). I use one dedicated user per externally accessible service.
Should the machine itself become compromised this would indicate that my personal master key and master password have been compromised or someone gained physical access. That would require me to restart from a blank page anyways.