Technical Summary: Adjust for Trust: Mitigating Trust-Induced Inappropriate Reliance on AI Assistance

Problem Statement

User trust acts as a critical modulator in AI-assisted decision-making, yet it is often miscalibrated relative to an AI system's actual capabilities. This misalignment leads to inappropriate reliance:

• Over-reliance: Occurs when user trust is excessively high, causing users to accept incorrect AI predictions (e.g., medical doctors accepting misdiagnoses 26% of the time when trust is high vs. 8% when lower).
• Under-reliance: Occurs when user trust is excessively low, causing users to reject correct AI predictions (e.g., doctors rejecting correct diagnoses 68% of the time when trust is low vs. 40% otherwise).

The paper posits that these extreme trust levels act as cognitive biases that hinder critical evaluation. Current AI assistants typically provide static behavior regardless of the user's evolving trust state. The core problem addressed is how to dynamically adapt AI behavior to mitigate these trust-induced biases and foster appropriate reliance.

Methodology

Experimental Design

The authors conducted a series of controlled user studies across two decision-making domains:

1. ARC Task: Laypeople answering grade-school science questions (39 filtered questions).
2. Diagnosis Task: Professional medical doctors diagnosing patients based on symptoms (55 cases from the DDXPlus dataset).

Interaction Flow:
For each of 30 sequential problems, users followed a three-stage process:

1. Independent Decision: User makes an initial decision.
2. AI-Informed Revision: User views AI prediction and confidence, then makes a final decision.
3. Trust Update: User observes ground truth and reports their trust in the AI (0–10 scale) for the next interaction.

AI Simulation:
The studies utilized a simulated AI assistant with two calibration profiles:

• Calibrated (ArcC, DiagC): Confidence scores matched prediction accuracy.
• Overconfident (ArcO): Confidence scores were higher than actual accuracy.

Trust-Adaptive Interventions

The paper investigates whether adapting AI behavior based on the user's reported trust level (at the start of an interaction) can improve outcomes. The interventions were triggered by thresholds (Trust < 5 for low; Trust > 8 for high):

1. Supporting Explanations (Low Trust): Providing natural language reasoning supporting the AI's prediction to encourage users to reconsider disregarding the advice.
2. Counter-Explanations (High Trust): Providing reasons why the AI's prediction might be incorrect (e.g., "It is possible that...") to act as a cognitive forcing function against blind acceptance.
3. Deceleration (High Trust): Inserting a forced 10-second pause before the final decision to promote deliberation.
4. Combined Strategy: Applying supporting explanations for low trust and counter-explanations for high trust within the same session.
5. LLM Generalization: Replicating the adaptive strategy using a real Large Language Model (LLaMA-3.1-8B) instead of a simulated agent.

Evaluation Metrics

• Switch Rate: Frequency of users changing their decision to match the AI.
• Over-Reliance: Switching to an incorrect AI prediction.
• Under-Reliance: Failing to switch to a correct AI prediction.
• Total Inappropriate Reliance: Sum of Over- and Under-Reliance.
• Final Decision Accuracy: Overall correctness of the user's final decision.

Key Contributions

1. Empirical Validation of Trust-Reliance Correlation: The study confirms a strong correlation between self-reported trust levels and reliance behavior. High trust correlates with increased Over-Reliance, while low trust correlates with increased Under-Reliance.
2. Trust-Adaptive Interventions: The authors demonstrate that dynamically selecting intervention types based on trust levels significantly outperforms static interventions:
   • Supporting explanations provided only during low trust reduce under-reliance without exacerbating over-reliance.
   • Counter-explanations provided only during high trust reduce over-reliance without increasing under-reliance.
3. Deceleration Efficacy: Forced pauses were found effective at reducing over-reliance during high-trust moments but ineffective at mitigating under-reliance during low-trust moments.
4. Generalization to LLMs: The findings hold when applied to a real LLM assistant, showing that trust-aware adaptation improves decision accuracy and reduces inappropriate reliance even without explicit control over model calibration.
5. Trust Modeling Limitations: The paper attempts to model user trust using interaction features (e.g., recent AI accuracy, confidence smoothing). Results indicate that current heuristic and learning-based methods are insufficient for accurately predicting moments of extreme trust (F1 scores ≤ 0.5), highlighting the difficulty of inferring trust solely from interaction logs.

Results

• Reduction in Inappropriate Reliance: The adaptive strategy (supporting explanations for low trust, counter-explanations for high trust) yielded up to a 38% reduction in inappropriate reliance and a 20% improvement in decision accuracy compared to non-adaptive baselines.
• Specific Gains:
  • In low-trust scenarios, adaptive supporting explanations reduced under-reliance by 13–31%.
  • In high-trust scenarios, adaptive counter-explanations reduced over-reliance by 10–23%.
  • The combined adaptive approach achieved the highest final decision accuracy (67.8% vs. 58.4% in the control condition) in the LLM study.
• Static vs. Adaptive: Providing explanations or counter-explanations always (regardless of trust) was less effective or even detrimental, often leading to over-reliance on the explanations themselves or a loss of trust in the system.

Significance and Claims

The paper claims that modeling and adapting to user trust is a viable and effective mechanism for improving human-AI collaboration. The significance lies in shifting the paradigm from static AI assistance to dynamic, trust-aware assistance.

• Mitigating Cognitive Bias: The work demonstrates that AI systems can act as "cognitive forcing functions" by strategically introducing friction (counter-explanations, pauses) or support (supporting explanations) to counteract the specific biases induced by a user's current trust state.
• Selective Transparency: The findings suggest that transparency (via explanations) should be modulated based on user trust. Full transparency is not always optimal; selective disclosure based on trust levels can balance the need for accountability with the need for appropriate reliance.
• Future Directions: The authors modestly note that while their threshold-based approach is effective, future work should explore more sophisticated criteria for intervention (e.g., accounting for user confidence and AI calibration) and better methods for inferring trust without explicit user reporting. They emphasize that these interventions are designed to support reasoning, not to manipulate users.

The study concludes that while trust estimation remains a challenging open problem, the mere ability to condition AI behavior on trust signals (even if self-reported) presents a promising avenue for reducing inappropriate reliance in high-stakes domains like healthcare and general decision-making.