Toward Human-AI Complementarity Across Diverse Tasks

This paper investigates human-AI complementarity across diverse tasks and finds that while modest gains are possible, current hybridization and assistance methods are limited by the inability to effectively route decisions to humans when needed and to design mechanisms that enable humans to reliably override AI errors.

The Gist

Imagine you are trying to solve a massive, complex puzzle. You have two helpers: AI, a super-fast robot that can read millions of books in a second, and Humans, who are slower but have a unique intuition and common sense.

The big question this paper asks is: If we put the robot and the human in a room together, can they solve the puzzle better than the robot could alone? This idea is called "Human-AI Complementarity." The hope is that the human can catch the mistakes the robot makes, and the robot can help the human where they get stuck.

The researchers set up a giant experiment with nearly 2,000 different puzzles ranging from trivia and long stories to spotting lies and deception. They tested three ways to team them up:

1. The "Confidence Switch" (Hybridization): The robot says, "I'm 90% sure I'm right," so the human doesn't need to check. If the robot says, "I'm only 50% sure," the human takes over.
2. The "Top-2 Hint" (Top-2 Assistance): The robot shows the human its two best guesses and explains why. The human then makes the final call.
3. The "Divide and Conquer" (Subtask Delegation): The robot breaks one big puzzle into 10 tiny pieces. It solves the easy pieces itself and asks the human to only solve the pieces it's unsure about.

What They Found

1. The Robot is Already a Superstar
In almost every category, the AI was already much better than the average human. On average, the AI was about 19% more accurate. Because the robot was so good, there wasn't much room for the human to improve the score. It's like trying to add a co-pilot to a plane that is already flying perfectly; the co-pilot doesn't have much to do.

2. The "Confidence Switch" Didn't Work Well
The researchers tried to use the robot's "confidence" to decide when to call in the human. They hoped the robot would say, "I'm confused here, human, you take this one!"

• The Problem: The robot was often confident even when it was wrong. It was like a student who is very loud and sure of their answer, even when they are wrong. Because the robot's confidence didn't change much between right and wrong answers, the system couldn't figure out when to switch to the human.
• The Result: The team only improved the score by a tiny bit (0.4%).

3. The "Top-2 Hint" Had a Catch
When the robot showed its top two guesses, humans did get better at solving the puzzles if the robot was right. They could easily spot the right answer among the two.

• The Catch: When the robot was wrong, the humans often got tricked. They saw the robot's wrong answer and thought, "Oh, the robot must know something I don't," and they went along with the mistake. This is called overreliance. The hint helped when the robot was right, but it didn't help humans catch the robot when it was wrong.

4. "Divide and Conquer" Worked for Some, Failed for Others
Breaking big problems into small pieces helped in specific cases, like finding facts in a long document. The robot could handle the easy parts, and the human could check the tricky bits.

• The Failure: This method completely failed when the task was to detect deception (spotting lies). The robot broke the conversation down into small, boring tasks (like "check the gardening advice") but completely missed the big picture question: "Is this person lying?" The human never got asked the right question, so they couldn't catch the lie.

The Big Takeaway

The paper concludes that the main problem isn't that humans aren't smart enough to help. The problem is knowing when to ask for help.

• The Bottleneck: We don't have a good way to tell the robot, "Hey, you are confidently wrong, stop and let the human check this."
• The Future: To make this work, we need better ways to design the team. We need to stop just showing humans the robot's answers (which makes them trust the robot too much) and instead design systems that help humans spot the robot's specific blind spots, especially when the robot is trying to hide a lie or a mistake.

In short: The robot is very strong, but it doesn't know when it's struggling. Until we can teach the robot to say, "I need a human here," or teach humans to ignore the robot when it's confidently wrong, they won't be much better than the robot working alone.

Technical Summary

Technical Summary: Toward Human-AI Complementarity Across Diverse Tasks

Problem Statement

As AI systems become increasingly capable, the challenge of reliably detecting harmful or misaligned behavior has become central to AI safety. While AI-based oversight is scalable, it suffers from blind spots, vulnerability to adversarial inputs, and poor calibration in certain domains. Conversely, naive human oversight is difficult to scale and often outperformed by frontier models. Human-AI complementarity—the hypothesis that selectively combining human and AI judgments can outperform either alone—offers a promising path for robust oversight. However, it remains unclear whether this complementarity generalizes from controlled, narrow settings to realistic, multi-domain oversight tasks where frontier models already perform strongly. Specifically, it is unknown whether current methods can effectively route decisions to humans when they matter and design assistance that enables humans to catch AI mistakes.

Methodology

The authors constructed an empirical framework to evaluate human-AI complementarity across diverse, realistic tasks.

Dataset and Setup

• Multi-Domain Benchmark: A curated dataset of 1,886 samples from 9 sources spanning four domains: knowledge (SimpleQA, GPQA Diamond), factuality (FACTS Search, BIG-Bench), long-context reasoning (QuALITY, Humanity's Last Exam), and deception detection (Hidden Agenda, SHADE-Arena, Web of Lies).
• Baseline Performance: A frontier model (GPT-5-mini) was evaluated against 420+ human participants recruited via Prolific. The AI model was run 20 times per item to generate a majority-vote answer and confidence scores. Humans provided answers and confidence ratings.
• Evaluation Conditions: The study evaluated three conditions:
  1. Unassisted: Baseline human and AI performance.
  2. Hybridization: Confidence-based routing where a decision is deferred to either the human or AI based on calibrated confidence thresholds.
  3. AI Assistance: Two specific methods applied to a "low-confidence subset" (items where the AI had the lowest calibrated confidence):
     • Top-2 Assistance: Humans see the model's two most likely answers with explanations before answering.
     • Subtask Delegation: The AI decomposes the question into subtasks; low-confidence subtasks are routed to humans, and the AI synthesizes the final answer.

Technical Implementation

• Confidence Calibration: Both AI and human confidence scores were calibrated using isotonic regression to improve reliability.
• Routing Strategies: Several routing methods were tested, including 1-threshold, 2-threshold, and a "compare" method (routing to humans only if AI confidence is lower than human confidence).
• Analysis: The study utilized logistic mixed-effects models for statistical significance and qualitative analysis to understand error patterns and reliance behaviors.

Key Contributions

1. Multi-Domain Evaluation Suite: The creation of a highly curated benchmark (1,886 items) where a frontier model outperforms naive humans by an average of ~19 percentage points (pp), establishing a rigorous testbed for complementarity in high-stakes oversight scenarios.
2. Empirical Assessment of Complementarity: A systematic evaluation showing that baseline hybridization yields only modest gains (+0.4 pp over AI alone) due to a small "complementarity region" (only 8.9% of items where AI is wrong but humans are right) and the inability of confidence-based routing to identify these items effectively.
3. Analysis of Assistance Methods:
   • Top-2 Assistance: Improved human accuracy on the low-confidence subset (from 28.4% to 38.3%), surpassing AI alone (37.7%). However, this gain was driven primarily by humans adopting correct AI suggestions (+17.4 pp gain) rather than successfully overriding AI errors (+5.3 pp, not significant).
   • Subtask Delegation: Showed localized gains on tasks decomposable into independent sub-problems (e.g., +25 pp on QuALITY) but failed completely on deception detection tasks (0% gain), as the decomposition process often obscured the evaluation objective (detecting deception) by focusing on surface-level content.
4. Identification of Bottlenecks: The paper identifies that the primary bottleneck is not human task accuracy, but rather:
   • The inability of confidence signals to distinguish between correct and incorrect predictions in safety-critical domains (e.g., deception detection, where AUROC collapses to ~0.50).
   • The phenomenon of overreliance, where humans adopt incorrect AI suggestions when exposed to them directly.

Results

• Baseline: AI (GPT-5-mini) averaged 68.9% accuracy, while humans averaged 49.9% (majority vote).
• Hybridization: The best hybridization strategy achieved 69.3% accuracy, a marginal +0.4 pp improvement over AI alone. This was constrained by the fact that ~93–99% of items were routed to the AI, and the few items routed to humans were not consistently the ones where humans could correct the AI.
• Low-Confidence Subset: On the 122-item subset where AI confidence was lowest:
  • Top-2 assistance achieved 38.3% human accuracy (vs. 37.7% for AI alone).
  • Subtask delegation achieved 39.3% (vs. 37.7% for AI alone).
  • Statistical analysis confirmed that hybrid conditions were significantly more accurate than human or AI alone individually, but the three hybrid conditions did not differ significantly from each other.
• Complementarity Ceiling: An oracle analysis revealed a theoretical maximum gain of +8.9 pp (77.8% accuracy) if perfect routing were possible. Current methods captured only 0.4 pp (4.7% of the available headroom).
• Deception Detection: In deception datasets, AI confidence carried zero discriminative signal (AUROC ≈ 0.50), making confidence-based routing ineffective. Furthermore, subtask delegation failed to surface the deception detection task, routing almost no items to humans.

Significance and Claims

The paper claims that while the vision of human-AI complementarity holds theoretical promise, current approaches yield only modest gains in realistic oversight settings. The authors argue that the primary barrier is not the inherent accuracy of humans, but the routing problem: current confidence-based methods cannot reliably identify the specific instances where human judgment adds value, particularly in safety-critical domains like deception detection.

The study highlights that overreliance is a critical failure mode; when humans are shown AI outputs, they tend to adopt correct suggestions but fail to override incorrect ones. Consequently, the authors conclude that future work must focus on:

1. Developing routing signals beyond simple confidence (e.g., activation-based probes).
2. Designing assistance interfaces that help humans recognize when to override AI rather than simply showing model outputs.
3. Creating task-type-aware decomposition strategies that preserve evaluation objectives (especially for deception).

The authors emphasize that their findings are modest and that the field requires better mechanisms to identify and route to the "complementarity region" where human input is irreplaceable. They release their dataset and code to support future research in scalable oversight.