“Crazy cat ladies” (formerly known as: witches, spinsters, old maids, among others) have historically been women who don’t marry, and prefer low-key companionship with cats (who demand a consensual relationship, and won’t tolerate being imposed upon) and other animals, as well as woman-to-woman friendships, over traditional domestic duties as a brood mare and household servant.
The whole hoarding thing is a trope, as it can happen to men and women alike, just like all hoarding tendencies. It may be slightly more common in women because women are raised to be more nurturing and compassionate, and often taught to ignore the practical boundaries of their efforts in pursuit of perfection, but that doesn’t mean it is exclusive in any way.
But either way, “crazy cat lady” is highly problematic language, and if you care at all about such things, you may want to avoid using it in the future.