Probably most important to your question is nutrients. Mold needs certain nutrients in order to grow, which is why it doesnt grow on everything. Some materials that have the nutrients mold needs include various foods, drywall, cotton, and some types of wood.
Mold also need decently specific moisture in its evironment (humidity of over 55%), a proper temperature range, shielding from UV rays that kill it, and if these conditions are met, mold can grow in about 24-48 hours.