I know people in the swim community who do this somewhat regularly (though usually the other way — Alcatraz to Aquatic Park).
There are big organized events (e.g., Alcatraz Invitational) as well as privately organized events. Given shipping lanes and the bay’s tides, it’s definitely not advisable to swim it without support in a boat, and ideally, radio contact with vessel traffic.