Siri Recap and Live Rewind Debut in watchOS Beta
I’m intrigued by these features but I definitely understand why people would be wary or completely untrusting of any company having something that is ostensibly “always listening to you. As Jason Snell explains below, that’s not really what’s happening. Even though everything is being transcribed that’s never what a user with this feature sees, it’s only ever a summary about what was discussed.
Apple went a long way to try to put things in terms that people can understand but it will take a LOT to get anyone to trust this. Even technical people that might read the white paper will be rightfully skeptical. Apple is pretty much the only company that I would trust with this kind of feature. They’ve shown time and again how serious they are about user privacy and never selling user data. Not to say they’re perfect or haven’t had scandals. Apple sells us products and doesn’t sell our data to advertisers or data brokers to make money.
Apple, obviously aware that this was going to be an issue, generated an entire Audio Intelligence white paper detailing how the features are secure and private. In general, audio is not recorded (per se), but buffered into a “secure exclave,” a portion of the watch’s chip that cannot be accessed by the system at large, and processed into text. That text is also encrypted and sent to the iPhone that’s paired with the watch, so only that iPhone can receive it and provide more sophisticated AI processing. And any cloud-based models used to generate summaries are part of Apple’s Private Cloud Compute infrastructure, which means they’re end-to-end encrypted and discarded after processing, so nobody (not even Apple) can see the data.
As 27.3 arrives, a few other details have emerged. Siri Recap text, which is based on conversations captured by the user by setting a time frame for capture or manually capturing via Control Center, is deleted after seven days if it’s not used. The user receives notifications on their watch when Siri Recap starts and stops, and a reminder is pushed to the user’s iPhone at the end of the day, pointing at the generated recaps, which are stored in a new Recaps tab in the Siri app.
The Apple Watch itself processes the audio with a “lightweight” ML model that detects if the audio it’s hearing is actually a conversation you’re a part of. (It really doesn’t want to attempt to recap other people’s nearby conversations. If you’re listening to audio on headphones, it takes that as a cue and doesn’t monitor any audio.)
If the audio passes that test, it’s encrypted and transferred to the paired iPhone, which turns it into “condensed text” (some filler words are removed). Then that text is encrypted and passed to Private Cloud Compute, which filters out sensitive personal information and other red-flag content and generates a summary, which is returned to the iPhone for display in that new tab. No content is attributed to specific speakers, and speakers aren’t even identified by name.
I think this feature could be really helpful and the summaries obfuscated enough to provide usefulness but not violate privacy of others or yourself. Time will tell but I’m planning on using this feature to see how things go and I’ll definitely turn it off as is appropriate or requested. I’ll share my thoughts as time goes on.