This isn’t a question of belief but of established nutritional practice. “Just eat fewer calories” isn’t a panacea, because people don’t respond in the same way to calorie deficits. Composition of diet, opportunity for exercise, age, physical health… you almost seem to hit on it when you talk about metabolism, but then you skitter right past when you conclude “Just eat more/less”.
It’s a rudimentary understanding of how people gain and lose weight, and it inevitably leads people towards crash diets and weight loss drugs that ruin your body in pursuit of a certain popular aesthetic.