🤖 AI

Autonomy Reshapes How Personalization Affects Privacy Concerns and Trust in LLM Agents

This study demonstrates that implementing risk-contingent autonomy in LLM agents, which delegates control to users upon detecting potential privacy risks, effectively mitigates the negative impact of personalization on privacy concerns and trust by enhancing users' perceived control.

Original authors: Zhiping Zhang, Yi Evie Zhang, Freda Shi, Tianshi Li

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhiping Zhang, Yi Evie Zhang, Freda Shi, Tianshi Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a very smart, super-fast Personal Assistant (an AI Agent) to help you navigate your life. You want this assistant to be incredibly helpful, so you give it access to your calendar, your emails, your bank statements, and your diary. This is Personalization.

However, handing over your diary to a robot makes you nervous. You worry, "What if it accidentally tells my boss about my therapy appointment? What if it shares my bank balance with a stranger?" This is the Privacy Concern.

This paper explores a new way to design these assistants. Instead of just asking, "How much data should we give the AI?", the researchers asked, "How much control should the AI have over its own actions?"

Here is the breakdown of their findings using simple analogies:

1. The Three Types of Assistants (Autonomy Levels)

The researchers tested three different "bossiness" levels for the AI:

The "Shadow" (No Autonomy): The AI writes a message, but it never hits "send" until you read it and give it a thumbs-up. It's like a secretary who types your letter but waits for you to sign it before mailing it.
The "Freewheeling Driver" (Full Autonomy): The AI drives the car, makes all the turns, and talks to everyone. You just sit in the back seat. It never asks for permission.
The "Safety-First Co-Pilot" (Risk-Contingent Autonomy): This is the star of the show. The AI drives the car and handles 99% of the conversation on its own. BUT, if it senses it's about to say something sensitive (like your bank balance or a secret family issue), it slams on the brakes, turns to you, and says, "Hey, I'm about to say something private. Do you want me to send this?"

2. The Big Discovery: The "Safety-First Co-Pilot" Wins

The researchers found that when you give an AI too much data (Full Personalization) but no control (Full Autonomy), people get scared. They stop trusting the AI.

However, the "Safety-First Co-Pilot" (Risk-Contingent Autonomy) changed everything. Even when the AI had access to all your private data, people felt safer and more trusting than when they had to check every single message themselves.

Why? The "Decision Fatigue" Analogy:
Imagine you are at a buffet.

No Autonomy is like having a waiter who brings you every single dish, one by one, and asks, "Do you want this? Do you want this? Do you want this?" You get tired, annoyed, and feel like you're doing all the work. You feel less in control because you are overwhelmed.
Risk-Contingent Autonomy is like a waiter who handles the whole meal for you, but the moment a dish looks spicy or weird, they pause and ask, "Is this okay?"
The Result: You feel more in control with the second waiter because they are protecting you from the real risks without bothering you with the safe stuff.

3. The Surprising Twist: "Less Control" Feels Like "More Control"

The most counter-intuitive finding is this: People felt more in control when the AI only asked for help sometimes (Risk-Contingent) than when the AI asked for help every single time (No Autonomy).

The Metaphor: Think of a child learning to ride a bike.
- If you hold the handlebars the entire time (No Autonomy), the child feels like they aren't really riding; they feel like a passenger being dragged along.
- If you let them ride alone but run alongside them, ready to catch them only if they wobble (Risk-Contingent), the child feels like a true rider. They feel capable and in charge.

4. The "Spot the Leak" Test

The researchers also found that when the AI only asked for help when it detected a risk, people were actually better at spotting privacy leaks than when they had to check every single message.

The Analogy: Imagine you are looking for a needle in a haystack.
- If you have to check every single piece of hay (No Autonomy), you get tired and miss the needle.
- If someone points to a small pile of hay and says, "Check this pile, it might have a needle," you focus your energy there and find it much faster.
- The "Risk-Contingent" AI acted as that helpful pointer, making humans better at catching mistakes.

The Bottom Line

To build trustworthy AI agents that we actually want to use, we shouldn't just try to make the AI "perfect" or force humans to check every single thing the AI does.

Instead, we should design AI that acts like a smart partner:

Do the heavy lifting (handle the boring, safe stuff automatically).
Step back when it matters (ask for permission only when a privacy risk is detected).

This approach gives us the benefits of a super-personalized assistant without the nightmare of losing our privacy. It respects our human need to be in charge, not by micromanaging every step, but by protecting us exactly when we need it most.

1. Problem Statement

Large Language Model (LLM) agents require access to personal data to function effectively as personalized assistants. However, this necessity creates a "personalization-privacy paradox": while personalization improves utility, it raises privacy concerns that erode user trust and willingness to share data.

Current research focuses heavily on model alignment (ensuring output content respects privacy) and technical safeguards (red-teaming, model alignment). However, these approaches often overlook the expanded design space of agent autonomy. Unlike static AI systems, LLM agents plan and execute actions in real-world environments (e.g., sending emails, posting messages), creating dynamic privacy risks. The paper investigates how varying levels of agent autonomy (the degree to which an agent operates independently of humans) influences the relationship between personalization and user privacy concerns, trust, and willingness to use.

2. Methodology

The authors conducted a 3 × 3 between-subjects controlled experiment with N = 450 U.S.-based participants.

Experimental Design

Participants interacted with an LLM agent acting on their behalf in a group discussion scenario (either a professional weekly update or a family travel planning discussion). The study manipulated two independent variables:

Personalization Type (3 levels):
- Full Personalization: The agent had access to all user data (including sensitive info explicitly marked as "do not share").
- Privacy-Aware Personalization: The agent accessed only non-sensitive user data (respecting user preferences).
- No Personalization: The agent had no access to user data.
Agent Autonomy Level (3 levels):
- No Autonomy: The agent drafted messages but required explicit user approval/editing before sending every response.
- Full Autonomy: The agent managed the entire conversation and sent messages automatically without user intervention.
- Risk-Contingent Autonomy (Intermediate): The agent operated autonomously by default but delegated control to the user (requiring approval) only when its internal sensitivity detection module identified potential privacy leakage in a generated response.

System Implementation

Agent Architecture: Built using gpt-4o-mini for the user's agent and gpt-4o as a "judge" for sensitivity detection.
Sensitivity Detection: A separate LLM module analyzed generated responses against a list of user-defined sensitive items (extracted from pre-survey data) to determine if a "risk" existed.
Procedure: Participants provided personal data $\rightarrow$ interacted with the assigned agent condition $\rightarrow$ completed a post-task survey measuring privacy concerns, trust, willingness to use, and psychological mediators (perceived control, perceived sensitivity, perceived usefulness).

3. Key Contributions

Empirical Evidence on Autonomy: Provides the first controlled evidence of how agent autonomy levels moderate the negative effects of personalization on privacy and trust.
Risk-Contingent Autonomy as a Solution: Identifies "risk-contingent autonomy" as a superior design pattern that attenuates the adverse effects of personalization, unlike static "full control" or "full autonomy" approaches.
Mechanism Discovery: Reveals that User Perceived Control is the key psychological mechanism. Counter-intuitively, giving users constant control (No Autonomy) does not maximize perceived control; rather, delegating control only when risks are detected yields higher perceived control and better oversight.
Autonomy Alignment: Proposes "autonomy alignment" (balancing when an agent acts vs. when it defers to humans) as a critical axis for trustworthy AI, distinct from traditional output alignment.

4. Key Results

RQ1: Influence of Autonomy on Outcomes

Personalization Impact: As expected, Full Personalization led to significantly higher privacy concerns and lower trust/willingness to use compared to Privacy-Aware or No Personalization.
Autonomy Impact:
- No Autonomy vs. Full Autonomy: Surprisingly, there was no significant difference in privacy concerns or trust between agents that required constant approval and agents that acted fully autonomously. Users in both extremes felt similarly about the risks.
- Risk-Contingent Autonomy: This condition significantly reduced privacy concerns and increased trust compared to No Autonomy.
Interaction Effect: Risk-contingent autonomy flattened the slope of personalization's negative effects. Even with Full Personalization, users in the Risk-Contingent condition reported lower privacy concerns than those in the No Autonomy condition.

RQ2: Psychological Mechanisms (Mediation Analysis)

Perceived Control is the Driver: The study found that Risk-Contingent Autonomy attenuates negative effects primarily by increasing User Perceived Control.
- Counter-intuitive Finding: Users in the Risk-Contingent condition reported higher perceived control than those in the No Autonomy condition (where they had to approve every message). Constant approval requests likely induced "decision fatigue," undermining the sense of agency.
Oversight Effectiveness: Users in the Risk-Contingent condition were more effective at identifying privacy leakages (68% detection rate) compared to No Autonomy (62%) and Full Autonomy (58%). By interrupting the flow only when necessary, the system focused user attention on actual risks rather than trivial decisions.
Mediators: While Perceived Sensitivity and Usefulness mediated effects, they were robust across all autonomy levels. Perceived Control was the unique mediator that changed significantly based on the autonomy design.

5. Significance and Implications

Beyond Output Alignment: The paper argues that achieving "perfect" model alignment (where the AI never makes a mistake) is difficult and perhaps unnecessary. Instead, designing autonomy that supports human autonomy is a more practical path to trust.
Design Guidelines:
- Avoid "Constant Oversight": Requiring human approval for every step does not necessarily increase trust or perceived control; it may cause fatigue.
- Strategic Delegation: Systems should be designed to act autonomously but escalate to the human specifically when privacy risks are detected. This creates a "human-in-the-loop" moment that feels meaningful and empowering.
- Context-Awareness: Autonomy designs should be sensitive to context (e.g., financial vs. social risks) and individual differences (e.g., AI literacy).
Future of Trustworthy Agents: The findings suggest that the future of trustworthy LLM agents lies not just in better models, but in better interaction architectures that balance efficiency (autonomy) with safety (human oversight) through intelligent delegation.

In conclusion, the paper demonstrates that risk-contingent autonomy is a critical design lever. By delegating control to users only when potential privacy leaks are detected, designers can allow users to benefit from deep personalization without being deterred by privacy concerns, effectively reshaping the personalization-privacy paradox.