Unless something has changed massively since I was deeply involved with this stuff, the people that sent you email may get a notification after some hours that their message is being delayed, and maybe after like 24-48 hours they might get a bounce. But if it’s just your SMTP server going down for an hour or two every now and then, the system should be able handle that seamlessly (barring some hiccups like messages showing up with timestamps hours in the past which sometimes is confusing).