It's definitely been life-changing for me. I never had the "touch". When I'd do stuff around the house, I'd put in screws at crazy angles and drill holes in the wrong places on walls. I never felt like I could fix or build anything. Machine tools let you build crazy shit without relying on your visual reasoning or coordination. You want a hole at a spot? Move the handwheels to that spot and you're there. There's no fucking around. That built up my confidence, and the stuff that does require spatial reasoning (like using a file to turn a round hole into a square one) happens slowly enough that you can make mistakes and still be okay.
If you decide to get into it, I'd recommend getting a lathe first. if you can't afford a mill, a drill press is still very useful. You can do almost anything on a lathe if you try hard enough, and a drill press makes some things a lot easier. Blondihacks will have a lot more to say about it, but I believe that's her recommendation.