This is more complicated, as is often the case. The phytoestrogen in soy does have similar effects on people as human estrogen, and the effects can be rather positive (regulation of weight and better insuline sensitivity). Source here.
The same review mentions potencial negative effects and concerns: “In adult male rats, exposure to dietary soy decreased androgen levels and prostate weight.” and “In humans, the use of soy or purified phytoestrogens in women at high risk of, or diagnosed with, breast cancer as well as in infants fed with soy-based formula are legitimate areas of concern.”
Both beneficial and adverse effects of soy seem to be understudied. For more information about soy phytoestrogens and both male and female fertility, check out this article.