In that I'm consistent: chess isn't a sport either. Sure, you feel wrecked after hours of concentration, but if that's the only criteria, then office jobs would be sport. If it has to be a game, then office politics could be treated as a game with rules and it would still be a sport.
So no, it has to have physical activity and the physical activity has to be significant enough to tire out muscles. Lifting a beer for a few minutes would be more of a sport than esports, chess or anything else majorly mental. The brain isn't a muscle.