Aside from situations involving false positive detections of wake words, there aren't other situations in which these devices are "listening". Watchdog groups have been eyeing these devices for years now, and there haven't been any confirmed cases of these things spying on their owners.