💻 computer science

Verification Without Distrust: Reframing User-Side Oversight as Routine Epistemic Governance in Everyday Human-Chatbot Interaction

This study challenges the assumption that user verification of AI outputs is driven by distrust, revealing through a mixed-methods analysis of 153 users that oversight functions as a routine form of epistemic governance distinct from trust, and proposing design directions for scaffolded oversight that enhance user satisfaction and agency.

Original authors: Aung Pyae

Published 2026-07-29

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Aung Pyae

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Trust Trap: Why We Check Our AI Friends Even When We Like Them

Imagine you are learning to ride a bicycle with a very helpful, super-smart training wheel that never falls over. In the world of science that studies how humans talk to computers (called Human-Computer Interaction), there is a long-held belief about how this relationship works. It goes like this: if you trust your training wheel completely, you won't need to look at it or check if it's wobbling. You just ride. This idea is called "trust calibration." It suggests that the more you trust a machine, the less you should worry about it, and the less you should double-check its work.

But here is the twist: what if that rule is wrong? What if, even when you love your smart assistant, you still feel the need to peek under the hood? This paper dives into that exact question. It looks at how people actually use chatbots—the AI tools that write our emails, solve our math problems, and help us study. The researchers wanted to know: Do people stop checking the AI's answers when they trust it? Or do they keep checking, no matter what? The answer changes how we should build these tools for the future.

The Great Double-Check Mystery

For a long time, experts thought of trust and checking as two ends of a seesaw. If you push down on "Trust," the "Checking" side should go up, meaning you stop verifying. The logic was simple: if you believe the AI is a good teammate, you shouldn't need to act like a detective. But the authors of this paper, Aung Pyae and their team, decided to test this theory with 153 frequent chatbot users. They asked these users simple questions: "How much do you trust the chatbot?" and "How often do you double-check its answers before you use them?"

The result was a total surprise. The data showed no detectable connection between trust and checking. It's as if the seesaw was glued flat. Whether a user said they trusted the chatbot a little or a lot, they checked the answers at the exact same rate. The researchers found a correlation of 0.01, which is basically zero. This means that even people who said, "I generally trust the chatbot's answers," were just as likely to double-check facts, numbers, and work-related content as people who were skeptical.

The paper explicitly rules out the idea that "better trust means less checking" in everyday chatbot use. It suggests that for frequent users, checking isn't a sign of distrust; it's just a habit. Think of it like a chef who trusts their favorite knife but still tastes the soup before serving it. The chef isn't doubting the knife; they are just doing their job. The authors call this "routine epistemic governance," which is a fancy way of saying: "It's just part of the daily routine to verify what you know."

Two Ways to Play with the AI

The study didn't just stop at checking. It looked at four different ways people interact with chatbot outputs:

Verification: Reading the answer and saying, "Yep, that looks right," without changing anything.
Refinement: Asking the chatbot to "make this shorter" or "explain it differently."
Correction: Telling the chatbot, "You got that math wrong, fix it."
Approval: Making sure the chatbot gets the "go-ahead" before it does something automatic.

The researchers discovered that these fall into two very different camps. Verification (just checking) is like a silent observer. It doesn't seem to make people happier or more satisfied with the chatbot, and it has no link to how much they trust the bot.

However, the other three—Refinement, Correction, and Approval—are the "interventionist" team. These are active moves where the user changes the chatbot's work. The study found that these actions are strongly linked to satisfaction. When users felt they could tweak, fix, or approve the chatbot's work, they enjoyed the experience much more. In fact, the data showed that people who did these active things were happier, even if they didn't necessarily trust the bot more. It's like the difference between watching a movie (checking) and directing the movie (intervening). The directors (interventionists) had a much better time.

The "Happy but Helpless" Gap

Here is the most interesting part of the story. The users in the study were very happy with their chatbots. Their satisfaction scores were high, averaging 4.22 out of 5. But when asked how much control they felt they had over the chatbot, the score dropped to 3.33.

The researchers call this the "Satisfaction-Control Gap." It's a big gap, with a statistical size of 0.72. Imagine you are eating a delicious meal prepared by a robot chef. You love the taste (high satisfaction), but you feel like you have no say in the recipe, no way to tell the chef to add less salt next time, and no idea how the food was made (low control). The study found that even when the chatbot did a great job, users didn't feel like they were the ones steering the ship. They felt like passengers, not drivers.

What Users Actually Want

When the researchers asked users what would make them feel better about using these chatbots, the answers were clear. Users didn't want the chatbot to just be "smarter" so they wouldn't have to check. They wanted tools to help them check better.

Specifically, they asked for:

Source Attributions: "Show me where you got that fact."
Process Transparency: "Show me your thinking steps, not just the answer."
Correction Persistence: "If I tell you to fix a mistake, remember that for next time."
Governable Memory: "Let me see and change what you've learned about my preferences."

The paper suggests that instead of trying to build chatbots so perfect that we stop checking them, we should build chatbots that make checking easy and useful. The goal isn't to remove the user's oversight; it's to support it.

The Takeaway

This paper flips the script on how we think about AI. It suggests that for people who use chatbots every day, checking the work isn't a sign that they don't trust the AI. It's a normal part of the process, like proofreading an essay. The real key to making people happy isn't just building trust; it's giving them the power to fix, refine, and approve the AI's work.

The authors conclude that we need to stop asking, "How do we build trust so users stop checking?" and start asking, "How do we support the checking that users are already doing?" By treating the user's need to verify and correct as a feature rather than a bug, we can build chatbots that are not just smart, but truly helpful partners.

Technical Summary: Verification Without Distrust

Problem Statement

Research on human–AI interaction has traditionally operated under a "reliance-calibration" paradigm, positing that trust and verification exist on a single inverse continuum: as trust in a system increases, the need for users to verify its outputs should decrease. This assumption, inherited from trust-in-automation literature, prescribes that design efforts should focus on building "appropriate trust" to render oversight unnecessary. However, this paper identifies a critical empirical gap: this inverse relationship has not been directly tested in the context of everyday, open-ended, and iterative human–chatbot interactions. Furthermore, the prevalence, structure, and outcomes of user-side "human-in-the-loop" (HITL) practices—specifically the spontaneous, self-directed work users perform to govern chatbot outputs—remain underspecified. The study challenges the implicit design goal of eliminating user checking, asking instead whether verification persists as a routine practice decoupled from trust levels.

Methodology

The study employed a cross-sectional mixed-methods design involving a survey of 153 frequent chatbot users. The sample was predominantly young (56.2% aged 18–24), student-enrolled (69.9%), and STEM-oriented (75.9% in Computing/IT or Engineering), with high usage frequency (67.3% daily).

Instrumentation:
The survey utilized a 5-point Likert scale across seven sections, including demographics, mental models, trust, oversight behaviors, perceived control, satisfaction, and design preferences.

Trust: Operationalized as a generalized dispositional orientation ("I generally trust the chatbot's answers").
Oversight Behaviors: Four distinct user-side practices were measured:
1. Verification: Double-checking information before acting.
2. Refinement: Asking the chatbot to rephrase, shorten, or improve answers.
3. Correction: Telling the chatbot when it is wrong.
4. Oversight (Approval): Confirming actions before automated execution.
Outcomes: Measured via task facilitation, enjoyment (satisfaction composite), and perceived control (felt agency).

Analysis:
Quantitative analysis included descriptive statistics, Pearson and Spearman correlations with Fisher-transformed 95% confidence intervals, and moderated regression. Qualitative data from open-ended prompts were analyzed via thematic analysis (single-coder). Sensitivity analyses were conducted to exclude ineligible participants (under 18 or non-users) and low-engagement respondents (identical Likert responses), confirming the robustness of the primary findings.

Key Results

1. No Detectable Trust–Verification Association (RQ1)

Contrary to the canonical prediction, the study found no detectable association between trust and verification ( $r = .01$ , 95% CI $[-.15, +.17]$ , $p = .90$ ). The confidence interval rules out moderate inverse linear relationships. This null result held across sensitivity analyses and proficiency subgroups. Verification is not a trust-contingent behavior that diminishes as trust grows; rather, it appears to be a routine practice maintained regardless of trust levels.

2. Distinction Between Evaluative and Interventionist Oversight (RQ2)

The study identified a structural heterogeneity in user-side HITL practices:

Evaluative Oversight (Verification): Shows no detectable association with trust and is only weakly tied to satisfaction ( $r = .21$ ).
Interventionist Oversight (Refinement, Correction, Approval): These practices show weak positive correlations with trust and are strongly tied to satisfaction (Refinement $r = .48$ , Oversight $r = .39$ , Correction $r = .38$ ).
Users widely endorse these interventionist practices (67–75% top-2 endorsement), suggesting that satisfaction in human–chatbot interaction is driven by the ability to actively shape outputs rather than passively receiving them.

3. The Satisfaction–Control Gap (RQ2)

A significant dissociation exists between task satisfaction and perceived control. Users reported high satisfaction ( $M = 4.22$ ) but substantially lower perceived control ( $M = 3.33$ ), resulting in a medium-to-large gap ( $d = 0.72$ ). Effective task outcomes do not entail a felt sense of agency. Regression analysis indicated that trust and oversight behaviors explain only a small portion of the variance in perceived control, suggesting that current interfaces fail to make user interventions "consequential" enough to generate a sense of steering.

4. Qualitative Insights

Qualitative findings revealed that users view chatbots primarily as instrumental tools rather than collaborators. Trust is conditional and failure-mode-specific (e.g., triggered by mathematical errors or fabricated references) rather than global. Verification is described as "routine epistemic hygiene" (analogous to proofreading) rather than a response to distrust. Users expressed a demand for epistemic infrastructure, specifically source attribution, reasoning transparency, and the ability to make corrections that persist across sessions.

Key Contributions

The paper makes five specific contributions:

Empirical Evidence: Demonstrates that among frequent users, verification shows no detectable association with trust, complicating the reliance-calibration paradigm.
Theoretical Distinction: Proposes a split between evaluative oversight (trust-decoupled, weakly tied to satisfaction) and interventionist oversight (weakly trust-correlated, strongly tied to satisfaction).
Satisfaction–Control Gap: Quantifies a gap ( $d = 0.72$ ) showing that effective outcomes do not produce felt agency, challenging the assumption that satisfaction implies control.
Theoretical Synthesis: Reframes user-side HITL as a "fast-process" complement to system-side HITL, organized around routine epistemic governance rather than trust calibration.
Design Directions: Derives four specific design directions grounded in user needs:
- Provenance cues: Embedded source attribution.
- Process transparency: Reasoning traces and uncertainty indicators.
- Correction persistence: Ensuring corrections propagate across turns/sessions.
- Governable preference memory: Inspectable, configurable, and reversible personalization.

Significance and Claims

The paper argues that the assumption "better-calibrated trust reduces checking" lacks empirical support in conversational settings. Instead, it posits that verification is a form of routine epistemic governance compatible with trust. The authors reframe the design question from "How do we build trust so users stop checking?" to "How do we support the checking users already do?"

The study concludes that user-side oversight should be treated as a first-class interaction mode. Rather than attempting to eliminate the need for user intervention through transparency or trust-building alone, design should focus on scaffolding the existing practices of verification, refinement, and correction. This approach acknowledges that effective task outcomes are distinct from felt agency and that meaningful human control requires making user interventions legible and consequential. The findings suggest that current chatbot interfaces, while satisfying, often fail to provide the "consequentiality" necessary for users to feel they are steering the interaction.