It is interesting. I think the negative perception here comes from the fact that ‘street smarts’ are often contrasted with ‘book smarts’. It is frequently expressed along with other anti-intellectual sentiments and is too often racially-tinged.
Calling a minority that in a ‘book-smart’ profession in the US is a tone-deaf party foul at best. I think the term ‘back-handed compliment’ is apropos.