It sounds like you are just going by elected presidents, but quite a few were multiple-term presidents, and those presidents had multiple elections with different opponents.
And sometimes, a losing opponent would go on to win a later election.
Also, no one ran against George Washington, twice.
Out of 59 elections (if you include Washington), I think there have been 49 white guys and one white lady who have lost a US Presidential election at least once (and may or may not have gone on to be President in a later election).