You can, the concern here is with people, not the specifications of SMS. People can be social engineered to give control of your phone number to someone else. It's happened before, it's not a hypothetical, and it's why security experts advise against using phone based methods.