Ehh…at least we know we don’t understand how the AI reached its conclusion. When you study human cognition long enough you discover that our beliefs about how we reach our conclusions are just stories the conscious mind makes up to justify after the fact.
“No, you’re doing it wrong” isn’t really a problem - it’s fundamental to most ML processes.