💬 NLP

ErrorMap and ErrorAtlas: Charting the Failure Landscape of Large Language Models

This paper introduces ErrorMap, a novel method for diagnosing the specific root causes of Large Language Model failures, and ErrorAtlas, a comprehensive taxonomy derived from analyzing 83 models across 35 datasets, to shift evaluation focus from mere success metrics to a deeper understanding of why models fail.

Original authors: Shir Ashury-Tahan, Yifan Mai, Elron Bandel, Michal Shmueli-Scheuer, Leshem Choshen

Published 2026-02-18

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shir Ashury-Tahan, Yifan Mai, Elron Bandel, Michal Shmueli-Scheuer, Leshem Choshen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher grading a stack of 100 math tests. You see that 20 students got the final answer wrong.

In the old way of doing things (traditional benchmarks), you would just write a big red "F" on the paper and say, "This student failed." You know that they failed, but you don't know why. Did they forget how to multiply? Did they misread the question? Did they run out of time? Or did they just have a bad day?

Without knowing the "why," you can't help them improve. You might spend hours teaching them multiplication when they actually just needed to learn how to read the instructions better.

This paper introduces a new way of grading called ErrorMap and a resulting "Atlas" (a map) called ErrorAtlas. Here is how it works, explained simply:

1. The Problem: The "Black Box" of Failure

Currently, we test AI models (like the ones powering chatbots) with standard tests. If the AI gets a question wrong, we just count it as a "failure." But a failure is like a flat tire on a car. You know the car stopped, but you don't know if it was a nail, a blowout, or a loose lug nut.

The authors argue that we need to stop just counting the flat tires and start figuring out what popped them.

2. The Solution: ErrorMap (The Detective)

ErrorMap is like a super-smart detective hired to look at every single wrong answer an AI gives. Instead of just saying "Wrong," it asks:

"Did the AI misunderstand the question?"
"Did it do the math right but forget to write the final answer?"
"Did it hallucinate a fact that doesn't exist?"
"Did it refuse to answer when it should have?"

It breaks down the AI's thought process step-by-step, finds the exact moment the train derailed, and labels that specific mistake.

3. The Result: ErrorAtlas (The Map of Mistakes)

After the detective (ErrorMap) analyzes thousands of mistakes from 83 different AI models across 35 different tests, it draws a giant map called ErrorAtlas.

Think of ErrorAtlas as a "Weather Map" for AI failures.

Instead of just saying "It's raining," the map tells you: "It's a heavy thunderstorm in the 'Math' region, but just a light drizzle in the 'Creative Writing' region."
It reveals that AI models often fail not because they are "dumb," but because they are bad at following specific instructions (like forgetting to list ingredients) or misinterpreting the intent of a question.

4. What Did They Find? (The Surprises)

The authors found some interesting "weather patterns" that nobody was paying attention to:

The "Missing Ingredient" Problem: A lot of AIs get the right idea but forget to include a specific detail the user asked for (like forgetting to list the inactive ingredients in a medicine). This is surprisingly common but rarely studied.
The "Wrong Question" Problem: Sometimes the AI answers the question perfectly, but it's answering the wrong question because it misunderstood what the human actually wanted.
Different Models, Different Flaws: Just like different car brands have different weak points, different AI models have different "failure signatures." One model might be great at math but terrible at formatting, while another is great at facts but bad at logic.

5. Why Does This Matter?

This is a game-changer for three groups of people:

For AI Builders (Developers): Instead of guessing what to fix, they get a specific repair manual. "Oh, the new version of the model fixed the math errors but made the formatting worse. Let's focus on formatting next."
For Test Creators (Benchmarkers): They can see if their tests are actually testing what they think they are. Maybe a test is supposed to measure "logic," but the AI is failing because of "bad formatting." Now they know to fix the test.
For Business Users: If you are a doctor choosing an AI to help diagnose patients, you don't just want the "highest score." You want to know: "Does this AI make fewer mistakes in medical facts?" ErrorAtlas helps you pick the right tool for the job.

The Big Picture

The paper quotes Aristotle: "It is possible to fail in many ways... while to succeed is possible only in one way."

Right now, we only look at the "one way" (success). This paper gives us a flashlight to look at the "many ways" (failure). By mapping out exactly how and why AI fails, we can stop building bigger, blurrier models and start building smarter, more reliable ones.

In short: We stopped just counting the wrong answers and started understanding the story behind the mistake. That's how we get better.

1. Problem Statement

Current Large Language Model (LLM) benchmarks primarily provide aggregate success metrics (e.g., accuracy scores) that indicate when models fail but not why. A single wrong answer on a reasoning dataset could stem from diverse root causes, such as:

Formatting errors.
Calculation mistakes.
Misinterpretation of instructions.
Dataset noise or ambiguity.
Weak reasoning capabilities.

Without disentangling these causes, benchmarks remain incomplete, hindering the ability to diagnose specific model limitations, guide targeted improvements, or make informed model selection decisions. Existing diagnostic methods often rely on input-based challenges (e.g., "count failures on differential equations") rather than analyzing the actual model output and the specific nature of the failure.

2. Methodology: ErrorMap

The authors introduce ErrorMap, an unsupervised, model-oriented framework designed to extract a unique "failure signature" for any LLM or dataset. It transforms raw, unstructured failure data into a structured, interpretable taxonomy. The process consists of two main stages:

Stage 1: Per-Instance Error Analysis

For every incorrect prediction in a benchmark:

Input: The model's incorrect output, the original prompt, reference answers (if available), and Informative Correct Predictions (ICPs) from other models in the benchmark. ICPs serve as rich reference points to approximate full solutions.
Process: An "analyst" LLM (judge) performs a step-by-step evaluation. It constructs a structured solution path, identifies required criteria/steps, and evaluates the incorrect response against them.
Output: The judge identifies the first major error that caused the failure (as subsequent steps are often flawed due to this initial mistake). It generates a structured summary and a short, free-form error label describing the specific skill failure (e.g., "Misapplication of General Position Formula").

Stage 2: Taxonomy Hierarchy Building

This stage organizes the thousands of generated error labels into a multi-level taxonomy through recursive iterations:

2.a. Error Categorization: Using a data mining approach (adapted from Wan et al., 2024), an LLM iteratively clusters unique error labels into broader categories. This involves:
1. Category Generation: Grouping initial labels.
2. Iterative Refinement: Updating categories with new batches of data.
3. Final Review: Ensuring coherence and removing ambiguity.
2.b. Error Assignment: The constructed taxonomy is applied back to all error instances to assign each specific error to the most appropriate category.
Result: A layered, hierarchical taxonomy (e.g., Computation Error $\rightarrow$ Formula Misapplication $\rightarrow$ Spatial Geometry Error).

3. Key Contributions

ErrorMap Framework: A novel, flexible pipeline to generate dedicated error taxonomies for any model or dataset combination without requiring manual labeling of the entire dataset.
ErrorAtlas: A static, comprehensive taxonomy of LLM failures derived from applying ErrorMap to 83 models across 35 diverse datasets (covering reasoning, medical, coding, function calling, etc.). It contains 17 high-level categories and numerous sub-categories.
Discovery of Under-explored Errors: The analysis reveals that prevalent failure modes like "Missing Required Element" (omitting mandatory details) and "Specification Misinterpretation" are currently under-studied in LLM research compared to hallucinations or reasoning errors.
Granular Model Profiling: The method demonstrates that different model families and versions exhibit distinct error patterns, allowing for nuanced behavioral profiling beyond simple leaderboard scores.
Open Source: The authors release the code, the ErrorAtlas taxonomy, and associated data for public use.

4. Key Results & Findings

Taxonomy Structure: ErrorAtlas identified 17 high-level error categories. The most prevalent were:
1. Missing Required Element (15.56%)
2. Specification Misinterpretation (11.5%)
3. Logical Reasoning Error (9.09%)
4. Incorrect Identification (8.98%)
5. Computation Error (8.45%)
Model-Specific Patterns:
- Gemini 2.0 Flash Lite showed the highest frequency of "Incomplete Content" errors but the fewest formatting errors.
- Claude 3.5 Haiku exhibited a higher tendency toward logical reasoning errors.
- Mixtral 8x22b was more prone to computation errors.
Benchmark vs. Skill Discrepancy: In challenging reasoning benchmarks (MMLU-Pro, GPQA, OmniMATH), approximately 44% of model errors were not due to weak reasoning but rather technical issues like computation errors, missing elements, or counting errors. This suggests that high-level benchmark scores often mask fundamental execution failures.
Version Comparison: Comparing Gemini 1.5 Flash vs. Pro revealed that the Pro version significantly reduced "Computation Errors" and "Incomplete Reasoning," validating specific architectural improvements.
Validation: The taxonomy achieved a 95.2% coverage score (only 1 error fell into "Other") and a 92% accuracy score when validated against a meta-judge. Robustness tests showed the taxonomy remains stable across different sampling sizes and model configurations.

5. Significance and Applications

Shift from Success to Failure: The paper advocates shifting the evaluation focus from "how well a model succeeds" to "why it fails," providing deeper insights into model limitations.
Debugging & Development: Model developers can use ErrorMap to detect regressions between versions (e.g., did a new update fix calculation errors but introduce formatting issues?).
Benchmark Curation: Benchmark creators can use the tool to validate if a dataset actually tests the intended skill or if it is plagued by noise and ambiguity.
Model Selection: Practitioners can select models based on specific domain needs (e.g., choosing a model with fewer "Factual Errors" for medical applications, even if its overall score is slightly lower).
Efficiency: By sampling only failed instances (approx. 10% of failures in the large-scale study), the method scales efficiently while maintaining high representativeness.

In conclusion, ErrorMap and ErrorAtlas provide a foundational shift in LLM evaluation, moving beyond opaque aggregate scores to a transparent, granular understanding of model failure modes, thereby enabling more targeted and effective model improvement.