No it’s not terribly stupid. Handicapped people already use other controllers without any issues.
A steering wheel is already electronically adjusted for speed. The servo will work just as well regardless of the controller device.
Along with lane assistance and other “self driving cars” it should be pretty evident that there is nothing dangerous about giving up the physical turny wheely kindergarten toy controller.