Others know a lot more about this then I do. The short answer is that some problems that are hard to solve now can be easily solved with quantum computers. Factorization of large integers is one of those things that is easily solved and public key crypto is based on that sort of problem. Hence the panic. Symmetric key crypto is not.
I think it comes down to just because quantum computers can solve some problems easily it does not mean they can solve all problems easily. Each situation needs to be looked at.