If you weren’t at a university it was generally a challenge to get hold of disks. Downloading at home took forever on a 28.8 or even 56k modem (ie. 56 kilobits per second).
Slackware and Redhat disk sets were the thing, in my experience. But generally that only gave you the compiled code, not the source (although there was an another set of disks with the source packages).
If you wanted to recompile stuff you had to download the right set of packages, and be prepared to handle version conflicts on your own (with mailing list and usenet support).
Recompiling the kernel with specific patches for graphics cards, sound cards, modems and other devices (I remember scanners in particular), or specific combinations of hardware was relatively common. “Use the source, Luke!” was a common admonition. Often times specific FAQ pages or howtos would be made available for software packages, including games.
XFree86 was very powerful on hardware it supported, but was very finnicky. See the other posts about the level of detail that had to be supplied to get combinations of graphics cards and monitors working without the appearance of magic smoke.
Running Linux was mostly a enthusiast/hobbyist/geek thing, for those who wanted to see what was possible, and those who wanted to tinker with something approaching Unix, and those who wanted to stretch the limits of what their hardware could do.
Many of those enthusiasts and hobbyists and geeks discovered that Linux could do far more than anyone previously had been prepared to admit or realise. They, and others like them, took it with them into progressively more significant, and valuable projects, and it began to take over the world.