Not insane at all. Child seats should be rear facing for quite a while and if the kid is asleep, they are not making any sounds. A big deviation from your routine can seriously fuck up remembering basic things. I personally have a mirror strapped to the rear headrest to avoid anything like that since I can see her every time I check my rear view mirror. But I’ve had people warn me how dangerous those are because it is an extra thing to break off in an accident. I’d rather take that risk than accidentally leave my child in a hot car.