Turbos spin far faster than (Roots-type) superchargers, and can therefore be much smaller.
Besides that, I don’t think rotating mass is really the issue. Yes, more inertia is like having a bigger flywheel so the engine will be slower to spin up/down, but that doesn’t consume much energy, especially in steady-state cruising.
Superchargers compress air - that takes energy. You then restrict it through the throttle body, because you’re not cruising with a wide-open throttle. That throws away all the compression.
You also have pumping losses and bearing/gear/belt losses.