Stallman wanted to use TRIX initially but it was considered too limited for the goals of GNU.
BSD was considered too but some of the Berkeley crowd were uncooperative because they secretly planned to make a commercial version (BSDi).
In the the end he compromised on Mach.
Thomas Bushnell:
RMS was a very strong believer – wrongly, I think – in a very greedy-algorithm approach to code reuse issues. My first choice was to take the BSD 4.4-Lite release and make a kernel. I knew the code, I knew how to do it. It is now perfectly obvious to me that this would have succeeded splendidly and the world would be a very different place today.
RMS wanted to work together with people from Berkeley on such an effort. Some of them were interested, but some seem to have been deliberately dragging their feet: and the reason now seems to be that they had the goal of spinning off BSDI. A GNU based on 4.4-Lite would undercut BSDI.
So RMS said to himself, “Mach is a working kernel, 4.4-Lite is only partial, we will go with Mach.” It was a decision which I strongly opposed. But ultimately it was not my decision to make, and I made the best go I could at working with Mach and doing something new from that standpoint.
This was all way before Linux; we’re talking 1991 or so.
From “The Daemon, the GNU and the Penguin” by Dr. Peter H. Salus.