Yes, you can actually have mold grow on your skin. But there are a lot of bacteria competing for food with the mold, as well as body oils and waste secretions, on top of the body’s immune system being pretty good at taking care of invasions.
There are quite a few different types of fungal skin infections (athlete’s foot, ringworm, fungal nails, an array of yeast infections), some can be pretty damn persistent!