Technical Summary: AI, Take the Wheel: What Drives Delegation and Trust in Human–Computer Cooperative Question Answering?

Problem Statement

Current research on human-AI collaboration often relies on post-hoc acceptance measures or synthetic tasks that fail to capture real-world reliance dynamics under uncertainty. A critical gap exists in understanding how skilled users make two distinct reliance decisions simultaneously: proactive delegation (deciding when to let an AI act autonomously without reviewing its output) and deliberative adoption (evaluating AI suggestions and deciding whether to accept them). Furthermore, prior work rarely studies these patterns together within the same realistic setting using the same users, leaving a lack of insight into how humans calibrate trust, which models they trust, and why.

Methodology

The authors conducted a competitive human-AI trivia tournament involving 23 expert human players and 16 distinct AI agents across 24 matches. The study utilized a "Quizbowl" format, a well-established academic trivia competition, to create high-stakes, time-pressured environments.

Experimental Design

The tournament featured two distinct phases corresponding to the two reliance decisions:

1. Tossup Phase (Proactive Delegation): Questions are read aloud with increasing ease. Players (human or AI) must "buzz" in immediately to answer. Teams can choose to mute an AI teammate, preventing it from buzzing for the remainder of the game. This captures the decision to rely on AI autonomy without oversight.
2. Bonus Phase (Deliberative Adoption): After a correct tossup, teams answer a three-part question. Humans first provide an initial guess without AI assistance. They then view suggestions from two AI teammates, including confidence scores and textual explanations. Finally, the team deliberates and submits a final answer. This captures the decision to adopt or reject AI output after evaluation.

Participants and Systems

• Humans: 23 experienced trivia players (mean competitive experience: 3.2 years) formed nine teams.
• AI Agents: 16 unique agents were developed through an open competition, utilizing diverse architectures (e.g., single-model calls, multi-step pipelines, ensemble voting) and base models (GPT-4.1, GPT-4o, Claude 3.5, DeepSeek V3, Cohere).
• Question Design: Questions were adversarially designed to exploit specific weaknesses in both humans (precise factual recall) and AI (culturally embedded reasoning), ensuring complementarity and preventing either party from dominating solo.

Data Collection

The study captured 387 delegation decisions (muting/buzzing) and 1,440 adoption decisions. Behavioral traces included agent selection, muting states, buzz timing, initial human guesses, AI outputs (answers, confidence, explanations), and final consensus answers.

Key Contributions

The paper introduces a novel benchmark for studying human-AI collaboration that decouples delegation and adoption decisions. It provides a comprehensive analysis of reliance patterns, identifying specific failure modes and design principles.

1. Asymmetric Reliance Errors

While human-AI teams outperformed both humans and AI alone (81.7% final accuracy vs. ~43% for humans alone), collaboration was not perfect. The study identified two primary error modes:

• Under-reliance (3.9%): Humans failed to adopt correct AI suggestions when their initial guess was wrong.
• Over-reliance (1.7%): Humans abandoned correct initial guesses to follow incorrect AI suggestions.
  Under-reliance was more frequent and severe than over-reliance.

2. Drivers of Trust and Miscalibration

• Confirmation Bias: When an AI agreed with a human's initial incorrect answer, under-reliance spiked to 64.5%. Humans trusted the agreement signal even when it reinforced a wrong judgment.
• Confidence Calibration: Model confidence scores were poorly calibrated. When humans and AI disagreed, relying on model confidence to select the correct answer performed near chance.
• Explanation Features: A significant gap exists between what predicts AI correctness and what humans trust.
  • Predictors of Correctness: LLM-assessed features like "Question Understanding" (76% accuracy) and "Reasoning Coherence" (72%).
  • Predictors of Human Trust: Surface signals like "Contains Quotes" (70% accuracy) and "Semantic Similarity" (66%).
  • The Bridge: Only "Evidence Grounding" (citing specific clues from the question) appeared in the top predictors for both correctness and human selection.

3. Learning and Adaptation

Teams demonstrated the ability to learn over time. Utilization rates (adopting correct AI answers) and discernment (selecting the right AI answer when models disagreed) improved significantly across rounds, particularly on difficult questions. Teams also learned to mute AI teammates strategically, capturing 79% of the potential points gain compared to an oracle policy, though they often muted too late.

Results and Design Principles

The study distills five design principles (DP) to improve human-AI collaboration systems:

1. Granular Control: Provide context-dependent toggles (by topic or difficulty) rather than binary on/off controls, allowing users to manage AI involvement strategically.
2. Standardized Confidence: Invest in cross-model calibration, especially for disagreement cases where users need the most help.
3. Accumulating Evidence: Surface context-specific analytics (e.g., "AI helped on 8/10 science questions") to accelerate trust calibration, rather than relying on static pre-deployment summaries.
4. Mutual Coverage: Design systems to target under-reliance by highlighting cases where AI confidence is high in domains where the user historically struggles.
5. Evidence-Grounded Explanations: Anchor explanations in observable input features (specific clues, quotes) rather than abstract reasoning. This is the only feature shown to predict both AI correctness and human trust.

Significance

The paper argues that improving human-AI collaboration requires moving beyond simple adoption metrics to understand the process of reliance. By studying experts in a domain where they possess genuine skill, the authors reveal that even skilled users suffer from systematic miscalibration, driven by confirmation bias and a mismatch between AI reasoning quality and human trust signals.

The study demonstrates that while collaboration is synergistic, it is not frictionless. The findings suggest that future systems must prioritize calibrated confidence, evidence-grounded explanations, and mechanisms that help users refine trust dynamically. The authors release the dataset, tournament platform code, and analysis scripts to support reproducibility and future research in high-stakes collaborative domains.