Must be regional or something else because I’ve never heard “old man” used in any other context than as a father. I’ve lived in Florida and the west coast and traveled all over the US for context.
“The ‘ol’ ball and chain” is used for spouse, unfortunately it’s usually used for women.