My house is very wel insulated. It doesn’t take much energy to keep it at a nice temperature in winter.
In summer though, it can get very hot inside. The reason is that I have some fairly large, south facing windows. And once it’s hot inside, it’s very hard to cool it down again.
I should really invest in some blinds, preferably outside, to keep the sun out during hot days.
But the point is that insulation and keep heat in/out is not perfectly symmetrical.