it is wrong to cast aspersions on someone because of something they did not choose.
Maybe it's not about casting aspersions but rather fitness for the job. If a surgeon loses their arms in a tragic accident they probably won't be allowed to operate. Pilots are forced to retire at a certain age because of this well documented age-related decline. If natural mental decline from age impairs one's ability to make fair and reasonable judgements and/or causes one to lose touch with the society they are resolving conflicts within, that seems like it would similarly impair a judge's ability to effectively do their job.