With 2, the natural numbers divide into equal halves. One of which we call odd and the other even. And we use this property a lot in math.
If you do it with 3, then one group is going to be a third and the other two thirds (ignore that both sets are infinite, you may assume a continuous finite subset of the natural numbers for this argument).
And this imbalance only gets worse with bigger primes.
So yes, 2 is special. It is the first and smallest prime and it is the number that primarily underlies concepts such as balance, symmetry, duplication and equality.