I never tried to keep track, but I always assumed each correctly identified lie/truth gets a point, and each mistake gives a point to the other team. Keep in mind that the show gets edited down and you don’t see everything that the audience did, while the score probably includes those things you didn’t see.
But like, absolutely no one watches this show for the score, so who cares? It might as well be QI’s scoring system :)