Your problem is you are viewing this as some sort of moral argument. I am not claiming landlords are sinners, Im saying that their class existing is harmful and land should not be commodified as it currently is.
Small landlords of the sort that you described are indeed just making long-term investments that are likely to yield a decent return or become a source of stability as an appreciating asset. It’s the kind of investment that we should want lots of people to be able to take advantage of.
Except by its very nature it is extractive. The renter is always getting fucked over in the situation.