Harry’s reaction alone is probably enough for Dumbledore to know something’s wrong, and asking Harry is just confirmation. Like you said, no reason to be aggressive to him.
I think Dumbledore overall is a brilliantly written character. His calmness portrays a deep wisdom and vast knowledge. The only time we’ve ever seen him outright angry is when Umbridge began to manhandle the student who sold out Dumbledore’s Army. He instantly touched upon a fury that made Umbridge and the Ministry realize they needed to calm down.
He really hasn’t expressed anger otherwise. He was calm facing off against Voldemort in Book 5. He was disappointed in the Dursleys for abusing Harry. He was apologetic when Harry lost control while grieving for Sirius.