When you see the cue that the event has happened, you rewind
The event has happened, or the aftereffects that the event happened. That is my point, the aftereffects matter as much as the event itself. As long as the ‘after’ looks different than the ‘before’ for any reason, that is a marker to give you an indication on which way to go, rewind, or advance.
And yes, either the effect or the aftereffects has to last long enough to be noticed by humans, less long by AI (faster to detect changes than humans). But the vast majority of events, when humans are involved, leave long aftereffects usually. Yes, not 100% of the time, but usually.