Indeed, and it turns out that in order to predict the next word these things may be thinking about stuff.
There's a huge amount of complex work that can go into predicting stuff. If you were to try to predict the next word that a person you're speaking with was going to say, how would you go about it? Developing a mental model of that person's thought processes would be a really good approach. How would you predict what the next thing that comes after "126+118=" is? Would you always get it exactly correct, or might you occasionally predict the wrong number?
I think you're starting from the premise that these things can't possibly be "thinking", on any level, and are trying to reinterpret everything to fit that premise. These things are largely opaque black boxes, just like human brains are. Is it really so impossible that thought-like processes are going on inside both of them?