Ah, going the George Bush route that the absence of evidence is not the evidence of absence. So, let me tell you about the unicorns on Mars. What, you can't prove they're not there.
Seriously though, we do have quite a few records from then since the Egyptian kings really wanted people to know about them. We also have archeological finds of where the workers lived. Also in them being given lavish tombs that regular Egyptians at the time didn't. This stuff is really not hard to look up.