Extra components mean more specific hardware to complete each task. This more specific hardware can process the same data often faster and with less power consumption. The drawback is cost, complexity and these compose are only good for that one task.
CPUs are great because they are multipurpose and can do anything, given infinite time and storage. This flexibility means it isn’t as optimised.
People are not creating custom code to solve their own problems. They are running very common applications, using very common libraries for similar functions. So for the general user specific hardware for encryption, video codecs, networking etc will reduce power consumption and increase processing speed in a practical way.