And because they are such black boxes, there’s the sector of Explainable AI which attempts to provide transparency.
However, in order to understand data from explainable AI, you still need domain experts that have experience in interpreting what that data means and how to make changes.
It’s almost as if any reasonably complex string of operations requires study. And that’s what tech marketing forgets. As you said, it all has to come from somewhere.