That could get rather annoying. Imagine if when you installed your distro everything was compiled from source (I apologise if you use Gentoo or LFS), it would take a lot longer.
Also people could still just upload a binary to various package managers (assuming the source is available).