Because you wash them often enough that they don’t get moldy, hopefully.
Yes, you can actually have mold grow on your skin. But there are a lot of bacteria competing for food with the mold, as well as body oils and waste secretions, on top of the body’s immune system being pretty good at taking care of invasions.