The real downside to SMR drives is “random” writes; adjacent tracks need to be re-written, and then their adjacent tracks, and that keeps going until the tracks adjacent to a write happen to be empty. It doesn’t matter much for long sequential writes (because adjacent tracks will be overwritten anyway). I think the re-writing process also hurts read performance for the host, but reads alone don’t cause rewriting.
If you need to reshape/resilver your array (grow, shrink, or change geometry), it’ll probably take weeks or months with an SMR drive compared to days for a CMR drive.