Though with weather its more egregious, we have a huge dataset to test weather predictions vs weather results against, to test the accuracy of models.
You’re right - and the predictions are quite accurate for 24-48 hours out. The thing is - if you say “there is a 90% chance of rain” then you would expect that 1 out of 10 times it would rain. If it rains all 10 times then your probability estimate was wrong (it was 100% chance of rain). At least over a large sample set.
Nate actually goes into this quite a bit in his book The Signal and the Noise.