Not All Trust is the Same: Effects of Decision Workflow and Explanations in Human-AI Decision Making

This study investigates how decision workflows, explanations, and user expertise influence human-AI trust, revealing that a two-step workflow does not necessarily reduce overreliance, that reported trust and behavioral reliance are distinct constructs, and that the effectiveness of explanations depends on the interaction between workflow design and user domain knowledge.

The Gist

Here is an explanation of the paper "Not All Trust is the Same," broken down into simple concepts with everyday analogies.

The Big Picture: The "Trust Tightrope"

Imagine you are walking a tightrope. On one side is Overtrust (blindly believing everything the AI says, even when it's wrong). On the other side is Undertrust (ignoring the AI completely, even when it's right).

The goal of this research is to help people walk that tightrope perfectly: trusting the AI when it's right, but ignoring it when it's wrong. The researchers wanted to see if changing how we interact with the AI (the "workflow") or adding explanations (telling the AI why it made a choice) would help us walk that line better.

---

The Experiment: The "Student Advisor" Game

The researchers set up a game where 300 people acted as university advisors.

• The Task: Look at a student's grades and background, then decide: "Will this student graduate successfully, or will they drop out?"
• The AI: A computer program gave a prediction for every student.
• The Twist: The AI wasn't perfect; it got about 27% of the answers wrong. The humans also weren't perfect; they got about 30% wrong. This meant the human and the AI had different strengths, making it a real test of teamwork.

The researchers tested four different ways to play the game:

1. The "Instant" Way (1-Step): The AI gives its answer immediately. You just say "Yes" or "No."
2. The "Think First" Way (2-Step): You have to make your own guess before you see what the AI thinks. Then, you can change your mind if you want.
3. The "Why" Factor: In half the games, the AI explained why it made its choice (e.g., "This student is at risk because they missed too many exams"). In the other half, it just gave the answer with no explanation.

---

The Surprising Findings

1. "Saying" vs. "Doing" are Different Things

The Analogy: Imagine a friend who says, "I love this new restaurant!" (Reported Trust). But when you actually go there, they refuse to order anything and eat a sandwich they brought from home (Behavioral Trust/Reliance).

The Finding: The study found that just because people said they trusted the AI on a questionnaire didn't mean they actually followed the AI's advice.

• Takeaway: You can't just ask people, "Do you trust this?" and assume they will act on it. You have to watch what they actually do.

2. The "Think First" Trick Backfired

The Analogy: Imagine a coach tells you, "Make your own play call before I tell you the strategy." The idea was that this would force you to think for yourself so you wouldn't blindly follow the coach.

• The Expectation: This "2-step" method should stop people from blindly following the AI when it's wrong.
• The Reality: It did the opposite! When people were forced to make a guess first, they actually became more likely to blindly follow the AI later, even when the AI was wrong.
• Why? It seems that once people made their first guess, they felt a psychological pressure to stick with it. When the AI disagreed, they didn't think, "Maybe I'm wrong." Instead, they thought, "The AI must be right, so I'll change my mind to match it." They became more reliant on the AI, not less.

3. Explanations Are a Double-Edged Sword

The Analogy: Think of an explanation like a map.

• If you are already lost and confused (low domain knowledge), a map might just make you more confused.
• If you are a local who knows the area well (high domain knowledge), a map helps you confirm your route.

The Finding:

• Explanations didn't help everyone equally. They only increased trust when used in the "Think First" (2-step) setup.
• In the "Instant" (1-step) setup, adding an explanation actually made people trust the AI less.
• Takeaway: You can't just slap an explanation onto any interface and expect it to work. It depends entirely on how the user is interacting with the system.

4. Knowledge Matters

People who felt they knew a lot about the subject (university systems) trusted the AI differently than those who didn't.

• Those with low knowledge felt less confident in the AI.
• Those with high knowledge felt more confident in the AI, but only if they were allowed to think first (2-step) and see an explanation.

---

The Bottom Line: What Should Designers Do?

The paper concludes that there is no "one-size-fits-all" solution for building trust in AI.

1. Don't just ask, "Do you trust me?" Watch what people actually do.
2. Be careful with the "Think First" rule. Forcing people to guess before seeing the AI might actually make them more likely to blindly follow the AI later, which is dangerous if the AI is wrong.
3. Context is King. An explanation that works in one type of app might hurt trust in another. Designers need to test their specific workflow, not just copy-paste features from other studies.

In short: Trust is complicated. It's not a single switch you can flip. It's a delicate dance between what people say, what they do, how much they know, and how the system is designed.

Technical Summary

Here is a detailed technical summary of the paper "Not All Trust is the Same: Effects of Decision Workflow and Explanations in Human-AI Decision Making."

1. Problem Statement

The central challenge in AI-assisted decision-making (AI-DSS) is achieving calibrated trust. Users must trust the AI when it is correct and resist it when it is wrong. Two primary failure modes exist:

• Overtrust/Overreliance: Accepting incorrect AI recommendations.
• Undertrust/Underreliance: Rejecting correct AI advice.

Previous research has proposed two main interventions to improve calibration:

1. Explanations: Providing reasons for AI predictions to help users verify accuracy.
2. Decision Workflows: Specifically, the 2-step setup (cognitive forcing), where users must make an initial decision before seeing the AI recommendation, compared to the standard 1-step setup (seeing the AI recommendation immediately).

However, existing literature presents contradictory findings regarding the effectiveness of these interventions. Furthermore, there is a lack of clarity on whether self-reported trust (subjective questionnaires) and behavioral trust (reliance/agreement rates) measure the same underlying construct, and how these metrics interact with decision workflows and user characteristics (domain knowledge, AI experience).

2. Methodology

Study Design:

• Type: Between-subjects online experiment ($N=300$).
• Design: $2 \times 2$ factorial design.
  • Factor 1: Decision Workflow: 1-step (AI suggestion shown immediately) vs. 2-step (User makes initial choice, then sees AI suggestion).
  • Factor 2: Explanations: Present (local feature importance explanation) vs. Absent (prediction only).
• Task Domain: A fictional university student support scenario. Participants acted as advisors deciding whether students were "on track to graduate" or "at risk of dropping out" based on academic and demographic data.
  • Rationale: This domain offers moderate stakes (higher than trivial, lower than medical) and allows for complementary knowledge between humans and AI.
• AI System: A Wizard-of-Oz simulation was used to strictly control accuracy.
  • Accuracy: Fixed at 73% (matching the pre-study human baseline of 70% to ensure complementary collaboration).
  • Explanations: Pre-computed rule-based local feature importance (highlighting 3 most relevant features in green/red).
• Participants: Recruited via Prolific, balanced by gender, with a majority possessing relevant domain knowledge (university context).

Variables:

• Independent Variables: Workflow (1-step vs. 2-step), Explanation (Yes vs. No), Domain Knowledge, Prior AI Experience.
• Dependent Variables:
  1. Self-Reported Trust: Measured via a modified Human-Computer Trust (HCT) questionnaire.
  2. Behavioral Trust (Reliance):
     • Agreement Rate: Overall acceptance of AI suggestions.
     • Overreliance: Rate of accepting AI suggestions when the AI was incorrect.
     • Switch Rate: Rate of changing one's initial decision when the AI disagreed (measured only in the 2-step group).

3. Key Contributions

1. Distinction Between Trust Constructs: The study empirically validates that self-reported trust and behavioral reliance are distinct constructs. They are only weakly correlated ($r \approx 0.25$), suggesting that asking users if they "trust" the system does not accurately predict whether they will actually follow its advice.
2. Evaluation of the 2-Step Workflow: The study challenges the prevailing hypothesis that a 2-step cognitive forcing workflow reduces overreliance. Contrary to prior studies, this setup increased overreliance in the specific context tested.
3. Interaction Effects: The research demonstrates that the effectiveness of explanations is not universal; it depends heavily on the decision workflow. Explanations increased reported trust in the 2-step setup but slightly decreased it in the 1-step setup.
4. Role of Domain Knowledge: Domain knowledge significantly influences reported trust but not necessarily behavioral reliance, and it interacts with the decision workflow (users with high domain knowledge trusted the 2-step setup more).

4. Key Results

A. Trust vs. Reliance (RQ1)

• Weak Correlation: Self-reported trust and agreement rate had a weak positive correlation ($r = 0.25$). Similarly, trust and switch rate were weakly correlated ($r = 0.25$).
• Strong Reliance Correlation: Agreement rate and switch rate were strongly correlated ($r = 0.70$).
• Implication: High agreement rates can occur even with low trust (if human intuition matches AI), and high reported trust does not guarantee behavioral reliance. Both metrics must be measured separately.

B. Effect of Decision Workflow on Trust (RQ2)

• No Main Effect: The decision workflow (1-step vs. 2-step) did not have a direct main effect on self-reported trust.
• Crossover Interaction (Workflow $\times$ Explanation):
  • In the 2-step setup, the presence of explanations increased reported trust.
  • In the 1-step setup, the presence of explanations slightly decreased reported trust.
  • Conclusion: The impact of explanations cannot be generalized across different workflow designs.

C. Effect of Decision Workflow on Reliance (Overreliance)

• Increased Overreliance: The 2-step setup significantly increased overreliance (accepting incorrect AI advice) compared to the 1-step setup ($\beta = 0.46, p = .001$).
• Explanation Impact: Explanations did not significantly reduce overreliance in either setup.
• Domain Knowledge: Users with low domain knowledge reported lower trust but did not differ in reliance behavior. Users with high domain knowledge reported higher trust in the 2-step setup.

5. Significance and Implications

Theoretical Implications:

• Trust-Calibration Paradox: The study highlights that "trust" is not monolithic. Relying solely on self-reports or solely on behavioral metrics provides an incomplete picture of human-AI collaboration.
• Context-Dependency: The efficacy of "cognitive forcing" (2-step) and explanations is highly context-dependent. The 2-step workflow, intended to reduce bias, may inadvertently increase reliance on incorrect AI if users feel their initial intuition was validated by the subsequent AI explanation.

Design Implications for AI-DSS:

• Avoid One-Size-Fits-All: Designers should not assume that forcing a 2-step workflow or adding explanations will universally improve trust calibration.
• Evaluation Strategy: Developers must evaluate both perceived trust (usability/acceptability) and behavioral trust (actual reliance) separately.
• Workflow Selection:
  • 2-Step Setup: Useful for diagnostic purposes to understand how human intuition complements AI predictions, but risky for deployment as it may increase overreliance on errors.
  • 1-Step Setup: May be more suitable for real-world applications where immediate decision-making is required, provided explanations are carefully designed.
• Adaptive Interfaces: Future systems should consider adaptive workflows (e.g., using 2-step only when user confidence is low) or allow users to control when they receive explanations.

Limitations:

• The study used a simulated AI (Wizard-of-Oz) with rule-based explanations, which may not capture the complexity of real-world model errors.
• The scenario was hypothetical, lacking real-world consequences, which may have affected engagement levels.

In conclusion, the paper argues that achieving calibrated trust requires a nuanced approach that considers the interplay between workflow design, explanation types, and user characteristics, rather than relying on a single intervention like the 2-step setup.