I think this is completely inaccurate depending on what time you are talking about. I would say Pol Pot was probably one of the most ardent communist of the 50’s, it was just a weird type of agrarian communism. And in the regions he controlled he did attempt to construct a classless agrarian socialist society.
He had denounced Marx and created a form of Feudalism. We do not consider the Nazis to be Socialist either. His “agrarian Communism” was an expliciy rejection of Marxism from the get-go, as his concept of deindustrialization goes directly against Marxism.
If you have nothing in common with Communism except the name, you have to justify why you believe yourself to be Communist. Rather than doing that, Pol Pot stopped pretending and denounced Communism altogether.