Local Is Not a Sufficient Privacy Boundary: Governing OS-Integrated On-Device AI
This paper argues that local execution alone is insufficient for privacy in OS-integrated AI and proposes a comprehensive, OS-centered framework featuring a threat model, risk taxonomy, architectural controls, and an audit rubric to ensure accountability through constrained information flow, bounded authority, and auditable governance.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
The Big Idea: "Local" Isn't a Magic Shield
Imagine you have a very smart, helpful assistant living inside your house (your phone or computer). For a long time, people thought: "If this assistant never leaves the house to talk to the outside world, it's perfectly safe."
This paper argues that this is a dangerous misunderstanding.
Just because the assistant stays inside the house doesn't mean your secrets are safe. The real privacy risk isn't where the assistant is working; it's what it is allowed to look at, what it remembers, and what it is allowed to do.
Think of it like a butler in your home:
- The Old View: "As long as the butler never leaves the front door, he can't tell anyone our secrets."
- The New Reality: Even if the butler stays inside, he might:
- Read your diary, your bank statements, and your medical notes all at once.
- Write a summary of your life that he keeps in a locked box forever.
- Decide to send a letter to your boss based on a private argument you had with your spouse.
- Change his rules tomorrow because the "house owner" (the tech company) updated the software.
The paper says that saying "it runs locally" is like saying "the butler never leaves the house." It answers one small question, but it ignores the much bigger question: Who is the butler working for, and what rules does he follow?
The Problem: The "Super-Butler"
In the past, apps were like separate rooms. A calendar app only saw your calendar. A mail app only saw your mail.
Now, operating systems (like iOS, Android, Windows) are building a Super-Butler that can walk through all the rooms at once. It can look at your emails, your photos, your location, your messages, and your screen all at the same time to help you.
The paper identifies six ways this Super-Butler can still mess up your privacy, even if it never calls the cloud:
- Access Expansion: The butler suddenly has a master key to every room, not just the one you asked him to enter.
- Context Collapse: The butler mixes your "work life" with your "family life." He might use a private family argument to draft a professional email, accidentally revealing your personal drama to your boss.
- Inference Amplification: The butler puts two harmless clues together to guess something sensitive. (e.g., "You are at the hospital + You are searching for 'divorce lawyer' = You are going through a divorce.")
- Persistence: The butler writes down a secret summary of your life and keeps it in a drawer forever, even after you asked him to forget it.
- Actionable Disclosure: The butler doesn't just know a secret; he acts on it. He might send a text or a file to the wrong person because he thought it was helpful.
- Governance Drift: The rules change overnight. The company updates the software, and suddenly the butler can read new things or keep secrets longer, without you knowing.
The Solution: A New Set of Rules
The authors propose a new way to check if these AI systems are actually safe. They call it a "Privacy-by-Architecture" framework. Instead of just asking "Is it local?", they ask four specific questions about the system's "governance" (who is in charge):
1. The Context Broker (The Gatekeeper)
Before the butler looks at anything, a Gatekeeper must decide: "Do you really need to see the medical records to answer this question about the weather?"
- The Fix: The system should only let the AI see exactly what it needs for that specific task, nothing more.
2. The Memory Manager (The Librarian)
After the task is done, a Librarian decides what gets kept.
- The Fix: The butler shouldn't keep a permanent file on your private thoughts unless you explicitly say, "Save this." If you delete a photo, the butler must delete the summary he made about it.
3. The Tool Broker (The Doorman)
If the butler wants to send an email or open a file, a Doorman checks the plan.
- The Fix: The Doorman asks, "Is this email safe to send? Does it contain private info from a different part of your life?" If the answer is no, the Doorman stops it.
4. The Release Governor (The Inspector)
Every time the company updates the software, an Inspector checks if the new version broke the privacy rules.
- The Fix: We need proof that the update didn't secretly give the butler more power.
How They Tested This (The "Report Card")
The authors didn't hack into Apple, Google, or Microsoft. Instead, they looked at the public documents these companies wrote about their new AI features (like Apple Intelligence, Android AICore, and Microsoft Recall).
They created a 4-Level Report Card to grade these features:
- Level 0 (Absent): No rules at all.
- Level 1 (Nominal): They say they have rules, but you can't see them or prove they work.
- Level 2 (Operational): You can see the rules working in the settings.
- Level 3 (Robust): You can prove the rules work with logs, tests, and independent checks.
The Results:
- Apple: Good at keeping things local, but their "Cloud Fallback" (when they send complex tasks to the cloud) needs more proof that they aren't leaking data.
- Google: Good at isolating apps, but it's hard to prove they aren't mixing data from different apps behind the scenes.
- Microsoft: Their "Recall" feature (which takes screenshots of your screen) is very powerful but creates a huge privacy risk because it stores so much history locally.
The Bottom Line
The paper concludes that "Local" is not a privacy guarantee.
Privacy isn't just about where the computer chip is sitting; it's about who controls the flow of information.
- If a system runs locally but reads everything, remembers everything, and acts on everything, it is not private.
- True privacy requires constrained information flow (only seeing what's needed), bounded authority (not doing things without permission), and auditable governance (proof that the rules are followed).
The authors want regulators, companies, and users to stop being fooled by the word "Local" and start asking: "Who is the Gatekeeper, and what are the rules?"
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.