I think the problem, is with how fast this is moving in regard to both software and hardware and how accessible something that could easily be weaponized to normal people.
That's the potential for trouble I can see, for example Auto-GPT with improvements on the tool itself and its dependencies could make for something pretty powerful that could be set loose with a very wide remit and absolutely no limit to what it will do, to achieve its goal.
Time will tell of course. But this is probably one of, if not the fastest moving topic in tech we've ever had.