It has to be just barely persuasive enough to trigger the hunger for impossible magic.
When I was a kid, I would look for Matrix in computer games. Some sandbox where you can make anything with any laws of physics and as complex as possible and without thinking and it would run fast. It was, ahem, year 2005, computer graphics wouldn’t look so realistic. Well, Second Life exists, but it’s not alive, real humans come there and they are.
Now when there’s a tool to make something appear close to right for AIs, there won’t be a shortage of such scams just like there isn’t with dick enlarging pills.
It might start to be considered stupid in the society, but in secret a lot of people not understanding the fundamentals will still fall for such scams. Just like with dick enlarging pills!