The idea is not actually about a man’s physical hands. It’s a metaphor for putting in the work. That could be volunteering, going to bat for your community, spending quality time with your kids/grandkids/family, working long hours to make sure your family has what it needs to survive, etc.
Yes, some men do manual labor and have rough hands, but OP isn’t saying that all men should do manual labor, just that they should all put in the “work” to make the world, their community, and their family’s lives better.