That’s partially right, various skills and adaptations are indeed required. Both female gorillas and giant tortoises average around 200 lbs and I’m sure you wouldn’t be surprised if I told you that one of those two has shown limited success in the tree-climbing department.
But if we focus back on weight within single animal types, bears in this instance, you’ll find a rather tongue-in-cheek answer as to why polar bears can’t climb trees. But if you read a little bit further, you’ll find in regards to other bears that
… there are some exceptions. We’ve already mentioned polar bears, but they are not alone without the ability to climb trees. Adult brown bears are also unable to climb trees, although this is because of their size rather than the lack of trees in their habitat.
So bears can and do become too heavy to climb trees. As far as I can tell, all able-bodied black bears can climb trees, but they max out at ~550 lbs, just shy of your 650 lb limit.
So, it’s very possible for a type of animal famous for a type of behavior to size out from being able to perform that behavior.