💬 NLP

GRACE: Step-Level Benchmark for Faithful Reasoning over Context

This paper introduces GRACE, the first human-annotated step-level benchmark with a data-driven error taxonomy for evaluating faithfulness in context-grounded reasoning, demonstrating that integrating step-level faithfulness signals into reinforcement learning significantly improves both model accuracy and reliability.

Original authors: Hoang Pham, Dong Le, Anh Tuan Luu

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hoang Pham, Dong Le, Anh Tuan Luu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher grading a student's essay. The student gets the final answer right, but if you look closely at their work, you see they copied a fact from a different book, made up a number, or twisted a logic rule to get there.

In the world of Artificial Intelligence (AI), this is a common problem. AI models often give the correct answer, but their "thought process" (called Chain-of-Thought) is full of hidden mistakes, lies, or guesses that don't match the source material they were given. Until now, we've mostly only checked the final answer, like a teacher who only looks at the grade at the top of the page and ignores the messy work underneath.

Enter GRACE: The "Step-by-Step" Detective

This paper introduces GRACE (Grounded Reasoning Assessment through Chain-of-Thought Evaluation). Think of GRACE as a new, super-strict grading system that doesn't just look at the final answer; it inspects every single step of the AI's reasoning to see if it stayed true to the source documents.

Here is how the paper breaks it down, using simple analogies:

1. The Problem: The "Magic Trick" of AI

When an AI solves a hard question, it writes out a list of steps. Sometimes, it looks like a perfect, logical chain. But the paper found that AI models often perform a "magic trick":

The Illusion: They might pull a fact from the text, but then twist it to fit a conclusion that isn't actually supported.
The Cheat: They might use their own "memorized" knowledge (like facts they learned during training) instead of the specific document they were supposed to read.
The Result: They get the right answer, but for the wrong reasons. It's like a student guessing the right answer on a math test because they saw the answer key, even though their math was wrong.

2. The Solution: A New "Report Card"

The authors created a massive dataset (a benchmark) called GRACE. They took 10 different AI models and asked them to solve questions from 4 different types of tests (like logic puzzles and reading comprehension).

Then, they acted like a team of expert editors. They went through every single sentence the AI wrote and asked:

"Did you actually say this in the text?"
"Is this a logical step, or did you just jump to a conclusion?"

They found that nearly half of the steps in these "correct" answers were actually unfaithful (lying or guessing). This proves that checking the final answer isn't enough; we need to check the steps.

3. The Two "Tracks" of Mistakes

The paper discovered that AI makes two distinct types of mistakes, like a car having two different kinds of engine failures:

Track 1: The "Logic Glitch" (GRACE-Inference)
- Analogy: Imagine a lawyer who understands the law but argues it backwards.
- What happens: The AI understands the rules but twists them. It might say, "If A causes B, then B must cause A" (reversed reasoning), or it might ignore a specific rule the text mentioned.
- Where it happens: Mostly in logic puzzles and exam questions.
Track 2: The "Fact Glitch" (GRACE-Grounding)
- Analogy: Imagine a tour guide who points to a building and says, "This is the Eiffel Tower," when it's actually the Statue of Liberty.
- What happens: The AI makes up facts, contradicts what the text says, or mixes up names (e.g., saying "John" did something when the text said "Jane").
- Where it happens: Mostly in reading comprehension where you have to find specific facts in long documents.

4. The "Training Gym" for AI

The paper didn't just build a test; they used it to train AI.

The Experiment: They took smaller AI models and taught them using the GRACE "report card." Instead of just telling the AI "You got the answer right," they said, "You got the answer right, but Step 3 was a lie. Don't do that."
The Result: These smaller models got much better. They started making fewer lies and fewer logic errors.
Reinforcement Learning: They also tried a method called "Reinforcement Learning," where the AI gets a "reward" not just for the right answer, but for being honest in its steps. This made the AI both smarter and more trustworthy.

5. The Big Takeaway

The paper concludes that current AI models have a lot of room to improve. Even the most advanced models are still "hallucinating" (making things up) inside their reasoning steps, even when they get the final answer right.

By using GRACE, we can:

Detect exactly where the AI is lying or making a logic error.
Categorize the error (is it a logic twist or a made-up fact?).
Train AI to be more honest, ensuring that when it gives an answer, the path it took to get there is actually supported by the evidence.

In short, GRACE is a tool to stop AI from "faking it till they make it" and force them to do the hard work of reasoning correctly, step by step.

Technical Summary: GRACE - Step-Level Benchmark for Faithful Reasoning over Context

Problem Statement

Current Chain-of-Thought (CoT) reasoning in large language models (LLMs) often produces traces that appear transparent but may silently deviate from source evidence, even when the final answer is correct. While existing methods can detect hallucinations at the response level or identify span-level errors, they treat the reasoning chain as a black box. Consequently, they fail to pinpoint where in the chain a failure occurs or what type of error it represents. Furthermore, there is a lack of resources providing step-level faithfulness judgments specifically for context-grounded textual reasoning, distinct from the process faithfulness (whether the trace reflects internal decision-making) or formal verification domains like mathematics and code.

Methodology

The GRACE Benchmark Construction

The authors introduce GRACE (Grounded Reasoning Assessment through Chain-of-Thought Evaluation), a step-level faithfulness benchmark constructed through a three-stage pipeline:

Data Curation and Trace Generation:
- Source Datasets: Four datasets spanning two reasoning domains were selected: MuSiQue and 2WikiMHQA (evidence-grounded multi-hop QA) and ReClor and LogiQA (deductive reasoning). A groundedness filter was applied to ensure questions could be answered solely from the provided context.
- Model Coverage: CoT traces were generated by 10 models (8 open-weight, 2 proprietary) ranging from 7B to 35B parameters.
- Scale: This process yielded 160,000 traces containing over 640,000 reasoning steps.
Bottom-Up Taxonomy Discovery:
- Instead of imposing a predefined taxonomy, the authors discovered error categories via unsupervised clustering.
- An LLM judge (Qwen3.5-27B) provided free-form critiques and tags for 30,000 unfaithful steps.
- These critiques were embedded and clustered (UMAP + HDBSCAN), resulting in 138 initial clusters.
- Human researchers manually consolidated these into 8 distinct error categories, organized into two evaluation tracks:
  - GRACE-Inference (Deductive Errors): Reversed Reasoning, Wrong Argument Reading, Rule Violation, Overreaching Claim.
  - GRACE-Grounding (Factual Grounding Errors): Groundedness Violation, Contradiction, Confusion, Evidence Neglect.
Data Split and Annotation:
- Training Set (GRACE-train): Derived from LLM consensus (majority voting among three judges) on high-agreement steps, comprising 6,915 traces (29,612 steps).
- Test Set (GRACE-test): Constructed from traces where LLM judges disagreed, representing high-difficulty cases. These were manually annotated by human experts, resulting in 437 traces (2,044 steps) with high inter-annotator agreement ( $\kappa = 0.82$ for faithfulness, $\kappa = 0.75$ for error category).

Evaluation and Experiments

The paper evaluates current models on the GRACE test set using two metrics:

Step F1: Binary classification of faithful vs. unfaithful steps.
Category F1: Macro-F1 for error categorization among unfaithful steps.

Additionally, the authors investigate the utility of step-level faithfulness signals in Reinforcement Learning (RL) pipelines, specifically using Group Relative Policy Optimization (GRPO) with a Process Reward Model (PRM) trained on GRACE-train.

Key Results

1. Benchmark Performance

Detection vs. Diagnosis: There is a significant gap between binary detection (Step F1) and fine-grained diagnosis (Category F1). Even the strongest model, Gemini-3.1-Pro, drops from 81.46 (Step F1) to 72.08 (Category F1). Open-weight models show an even wider gap, indicating they can detect errors but struggle to classify them.
Model Capability: Performance generally correlates with model size, but proprietary models do not guarantee superior context faithfulness (e.g., GPT-4o-mini exhibited the highest error rate at 58.1%).
Fine-Tuning Impact: Small models (e.g., Qwen3-4B) fine-tuned on GRACE-train achieved performance comparable to or exceeding larger zero-shot models, particularly in error categorization. This suggests that explicit training on the error taxonomy is more effective than relying on prompt-based inference.

2. Analysis of Reasoning Failures

Correct Answers Mask Unfaithful Reasoning: Approximately 49.5% of traces containing at least one unfaithful step still reached the correct final answer. This disconnect confirms that output-level accuracy is insufficient for evaluating reasoning quality.
Track Differences: GRACE-Grounding datasets exhibited roughly 1.5x more unfaithful steps than GRACE-Inference datasets, likely due to the complexity of chaining facts across multiple passages versus single-passage deduction.
Inference Strategy: Providing the error taxonomy explicitly improved detection performance. However, the optimal inference strategy (all-steps vs. step-by-step) varied by model architecture.

3. Reinforcement Learning with Process Rewards

Integrating step-level faithfulness signals into RL training (F1 + PRM) yielded dual benefits:

Improved Accuracy: Downstream task F1 scores increased compared to training with F1 rewards alone.
Improved Faithfulness: The "Faith" score (average step-level faithfulness) increased, resolving the tension where accuracy-focused training sometimes degraded reasoning quality.

Significance and Contributions

The paper claims three primary contributions:

GRACE Benchmark: The first human-annotated, step-level faithfulness benchmark for context-grounded textual reasoning, featuring a high-quality evaluation set and a data-driven error taxonomy.
Data-Driven Taxonomy: An empirical taxonomy discovered via unsupervised clustering, distinguishing between GRACE-Inference (deductive/logical failures) and GRACE-Grounding (factual/evidence failures).
Benchmarking and RL Integration: The evaluation reveals substantial headroom for current models, demonstrating that step-level faithfulness assessment is necessary to surface errors concealed by outcome metrics. Furthermore, the work demonstrates that integrating context faithfulness signals into RL training pipelines improves both downstream task accuracy and reasoning reliability.

The authors conclude that step-level faithfulness evaluation remains an open problem, particularly for smaller models, but that fine-tuning on structured faithfulness data and incorporating process rewards can significantly enhance model performance and reliability.