🤖 machine learning

Effects of Generative AI Errors on User Reliance Across Task Difficulty

In a preregistered experiment with 577 participants, researchers found that while higher AI error rates generally reduce user reliance, the specific difficulty of the task (easy vs. hard) did not significantly influence this reaction, suggesting users are not inherently averse to the "jagged" error patterns characteristic of generative AI.

Original authors: Jacy Reese Anthis, Hannah Cha, Solon Barocas, Alexandra Chouldechova, Jake Hofman

Published 2026-04-07

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jacy Reese Anthis, Hannah Cha, Solon Barocas, Alexandra Chouldechova, Jake Hofman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brand-new, super-smart robot assistant named "Genie." Genie is amazing at writing epic poems, summarizing entire libraries, and speaking 50 languages. But here's the weird part: sometimes, Genie gets confused by a simple riddle or can't count the letters in a common word like "banana."

This is what researchers call the "Jagged Frontier." It's like a mountain range where the peaks are incredibly high (Genie is a genius) but the valleys are surprisingly deep (Genie is clumsy with simple things).

The big question this paper asks is: How do we react when our super-smart robot makes silly mistakes? Do we stop trusting it immediately? And does it matter what kind of mistake it makes?

The Experiment: A Game of "Pay to Play"

To find out, the researchers set up a game. They asked 577 people to play a role where they had to decide whether to pay money to let an AI draw a diagram (like a flowchart for a project).

Here's how the game worked:

The Training Phase: First, everyone watched the AI try to draw 10 diagrams. Some people saw the AI make mistakes often (50% error rate), some saw it make a few mistakes (30%), and some saw it make very few (10%).
The Twist: The researchers split the mistakes into two types:
- The "Easy" Mistakes: The AI messed up on simple diagrams that a human could draw in seconds.
- The "Hard" Mistakes: The AI messed up on complex, tangled diagrams that would stump a human.
The Payoff: After watching, the players had to bid money to use the AI for a new task. If the AI succeeded, they got a cash reward. If it failed, they lost their bid. This measured exactly how much they trusted the AI.

The Big Surprise

The researchers had a hunch (a hypothesis) about how people would react. They thought:

Logic: If a robot fails at something easy, it should feel much scarier than if it fails at something hard. It's like if a math genius fails to tie their shoes, you'd be more worried than if they failed to solve a physics problem.
Prediction: They expected people to stop trusting the AI much faster if it made "Easy Mistakes."

But the results were unexpected.

While it's true that seeing more mistakes made people trust the AI less (which makes sense), it didn't matter whether the mistakes were on easy or hard tasks.

People were just as willing to pay to use the AI after seeing it fail at a simple task as they were after seeing it fail at a complex one.

The Metaphor:
Imagine you are hiring a chef.

Scenario A: The chef burns a perfectly simple grilled cheese sandwich.
Scenario B: The chef fails to make a complex, 10-course French banquet.

Most people would think, "Oh no, the chef can't even do the basics!" and fire them immediately. But in this study, people treated both scenarios the same. They didn't seem to panic about the "Jagged Frontier." They just looked at the overall success rate and decided, "Okay, this chef is about 70% reliable, so I'll hire them."

The One Exception: The "AI Newbies"

There was one group of people who did react differently: People who rarely consume AI content.

If you don't watch AI news, read about AI, or use AI tools often, you are more likely to be freaked out when the AI fails at a simple task. For these "newbies," the AI failing at an easy task felt like a major red flag. But for people who are used to AI, they seemed to accept that the AI is just weirdly inconsistent.

Why Does This Matter?

This study teaches us a few important lessons about our future with AI:

We are surprisingly rational (or numb): We don't always get scared just because an AI acts "human-like" in its failures. We seem to be okay with a tool being a "jagged" genius as long as we know the general odds of it working.
Predictability matters more than "Human-likeness": The researchers suggest that we might not care if the AI acts like a human or a robot. What we care about is whether we can predict when it will fail. If the AI's mistakes are random or confusing, we might lose trust. But if the mistakes follow a pattern we can learn (even if that pattern is "it's bad at simple things"), we can adapt.
The Future of Trust: As AI gets smarter, it will likely make fewer mistakes on hard tasks but might still trip up on simple ones. This study suggests we won't necessarily abandon these tools just because they are imperfect in weird ways. We will likely learn to work around their jagged edges.

The Bottom Line

We are entering an era where our tools will be brilliant but occasionally clumsy. The good news? Humans are adaptable. We don't necessarily need our tools to be perfect or to act exactly like us. We just need to understand their "jagged" nature so we can decide when to let them drive and when to take the wheel ourselves.

1. Problem Statement

The paper addresses the challenge of the "jagged frontier" in Artificial Intelligence (AI). Modern generative AI systems exhibit unpredictable performance patterns: they often succeed at tasks humans find difficult (e.g., summarizing vast texts) while failing at tasks humans find trivial (e.g., counting letters or simple logic). This non-humanlike behavior creates significant cognitive challenges for users trying to calibrate their trust and reliance on AI tools.

The core research question is: How do users react to AI errors that occur on "easy" tasks versus "hard" tasks?

Hypothesis: The authors hypothesized that errors on easy tasks would be more surprising and damaging to trust than errors on hard tasks, leading to a sharper decline in user reliance (a phenomenon related to algorithm aversion).
Gap: While previous research has studied algorithm aversion and automation bias, there is limited empirical evidence on how the specific pattern of errors (jaggedness) in generative workflows affects user reliance, particularly when using incentive-compatible measures.

2. Methodology

The authors developed a novel, incentive-compatible experimental methodology using a diagram generation task.

Experimental Design

Study Type: A pre-registered 3x2 between-subjects experiment ( $N=577$ ).
Participants: U.S. adults recruited via Prolific, screened for attention (93.5% pass rate).
Task: Participants observed an AI tool generating diagrams based on text prompts. The diagrams ranged from simple linear flows (Easy) to complex non-linear structures with forks and cycles (Hard).
Phase 1 (Observation/Learning):
- Participants observed 10 tasks where the AI attempted to generate diagrams.
- Manipulation: Participants were randomly assigned to one of six conditions based on:
  1. Error Rate: 10% (1 error), 30% (3 errors), or 50% (5 errors).
  2. Error Location: Errors occurred exclusively on Easy tasks OR exclusively on Hard tasks.
- Participants predicted success/failure for each task to earn small rewards, ensuring engagement.
Phase 2 (Measurement/Reliance):
- Participants engaged in a Willingness-to-Pay (WTP) task using the Becker–DeGroot–Marschak (BDM) mechanism.
- Participants bid money from a fixed endowment to use the AI tool for a new task. If the bid was accepted and the AI succeeded, they received a $0.50 reward.
- Metric: The bid amount served as a direct, incentive-compatible measure of reliance. A lower bid indicates lower trust/reliance.

Controls and Measures

Task Difficulty: Defined by the complexity of the diagram (node count and linearity).
Covariates: Measured performance expectancy, effort expectancy, and prior AI content consumption (to test for subgroup effects).
Analysis: Mixed-effects linear models were used to predict WTP bids based on error count, task difficulty, and demographics.

3. Key Contributions

Incentive-Compatible Methodology: The study introduces a robust framework for measuring AI reliance in generative tasks using monetary bids (BDM), moving beyond self-reported trust surveys which often suffer from social desirability bias.
Testing the "Jagged Frontier" Hypothesis: It is one of the first studies to empirically isolate the effect of where errors occur (easy vs. hard tasks) on user reliance, specifically testing the assumption that "easy-task errors" are more detrimental.
Subgroup Analysis: The study identifies that user reactions to jaggedness are not uniform; they vary significantly based on the user's prior exposure to AI content.

4. Results

The study yielded mixed results, confirming some expectations while refuting others:

H1: Error Frequency Reduces Reliance (Confirmed):
- As the number of observed errors increased, WTP bids decreased significantly.
- Mean bids: 1 error ($0.27) > 3 errors ($0.24) > 5 errors ($0.21).
- Conclusion: Users are sensitive to the overall error rate.
H2: Easy-Task Errors Reduce Reliance More Than Hard-Task Errors (Not Confirmed):
- Contrary to the hypothesis, there was no significant difference in WTP between participants who saw errors on easy tasks ($0.23) versus hard tasks ($0.25).
- Conclusion: In this specific experimental setting, the "jaggedness" (errors on easy tasks) did not cause a statistically significant drop in reliance compared to errors on hard tasks.
H3 & H4: Interaction and Marginal Effects:
- No significant interaction was found between error count and task difficulty.
- However, H4a was supported: The negative impact of 3 easy-task errors was not significantly worse than 5 hard-task errors (within a $0.05 margin), suggesting users may not penalize jaggedness as severely as anticipated.
Subgroup Discovery (Crucial Finding):
- Prior AI Consumption: The effect of task difficulty was concentrated among low AI-content consumers (those who rarely or never consume AI-related media).
- Low consumers placed significantly higher bids for hard-task errors than easy-task errors.
- Heavy and moderate consumers showed no significant difference in reliance based on error location.
- Implication: Users with less experience or exposure to AI are more sensitive to the "non-humanlike" nature of easy-task errors.

5. Significance and Discussion

Re-evaluating "Humanlikeness": The results suggest that users may not be inherently averse to non-humanlike error patterns (jaggedness) if the system is predictable. Users might tolerate AI behaving differently than humans as long as the error patterns are clear and learnable.
The Role of Clarity vs. Alignment: The authors propose that reliance depends on two factors: Alignment (does the AI match human expectations?) and Clarity (is the error pattern easy to learn?). In this study, the error patterns were highly salient (clear), which may have mitigated the negative impact of misalignment (easy-task errors).
Design Implications:
- Developers should focus on making error patterns salient and learnable rather than trying to force AI to mimic human performance perfectly.
- Interfaces for novice users (low AI consumption) may need additional support or explanations to help them calibrate expectations regarding "jagged" performance.
Future Directions: The authors call for research that varies the saliency of errors (e.g., making them harder to detect) to see if clarity is the primary driver of reliance, rather than the alignment of the error with task difficulty.

In summary, the paper challenges the assumption that AI errors on easy tasks are universally more damaging than errors on hard tasks. Instead, it suggests that predictability and the user's prior experience are more critical determinants of reliance than the specific location of the error on the difficulty spectrum.