It’s entirely possible! I remember listening to a podcast on AI, where they mentioned someone once asked the questions “which mammal lays the largest eggs” to which the ai responded with elephants, and proceeded to argue with the user that it was right and he was wrong.
It has become a lot easier as I’ve learned how to kind of coach it in the direction I want, pointing out obvious errors and showing it what I’m really looking to do.
Ai is a great tool, when it works. As the technology improves I’m sure it will rapidly get better.