💻 computer science

PRISM: Probing Reasoning, Instruction, and Source Memory in LLM Hallucinations

The paper introduces PRISM, a comprehensive benchmark that reframes LLM hallucination evaluation as a diagnostic problem by disentangling errors into four specific dimensions across three generation stages, revealing critical trade-offs in model performance and providing a framework for developing more trustworthy AI systems.

Original authors: Yuhe Wu, Guangyu Wang, Yuran Chen, Jiatong Zhang, Yutong Zhang, Yujie Chen, Jiaming Shang, Guang Zhang, Zhuang Liu

Published 2026-04-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yuhe Wu, Guangyu Wang, Yuran Chen, Jiatong Zhang, Yutong Zhang, Yujie Chen, Jiaming Shang, Guang Zhang, Zhuang Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a brilliant but slightly confused assistant named "The Model." You ask it to write a story, solve a math problem, or give you medical advice. Sometimes, it does a great job. Other times, it confidently makes things up, ignores your instructions, or gets the logic wrong. This is called a hallucination.

Until now, when the Model messed up, we just said, "You got it wrong," and gave it a bad grade. But we didn't know why it failed. Was it because it forgot a fact? Did it misunderstand your rules? Or did it just get confused while thinking?

The paper PRISM introduces a new way to test these AI models. Instead of just looking at the final answer, PRISM acts like a medical diagnostician or a mechanic who opens up the engine to see exactly which part is broken.

Here is a simple breakdown of how PRISM works, using some creative analogies:

1. The Four Types of "Sickness"

PRISM breaks down every mistake the Model makes into four distinct categories. Think of these as four different types of errors a student might make on a test:

Knowledge Error (The "Fake Fact"): The Model thinks it knows the answer, but it's wrong.
- Analogy: Imagine a history teacher who confidently tells you that the Titanic sank in 1912 (it did), but then claims it was built by a company called "Titanic Corp" (it wasn't). The Model has the wrong data stored in its brain.
Knowledge Missing (The "Blank Page"): The Model doesn't know the answer at all, but it tries to guess anyway.
- Analogy: You ask the Model about a news event that happened yesterday (after it stopped learning). It doesn't have that info, but instead of saying "I don't know," it invents a story. It's like a student guessing on a test question they've never seen before.
Reasoning Error (The "Logic Glitch"): The Model has the right facts, but it can't put them together correctly.
- Analogy: You give the Model the ingredients for a cake (flour, eggs, sugar) and the recipe steps. It has all the right pieces, but it decides to bake the eggs before mixing them, or it forgets to turn on the oven. The logic chain is broken.
Instruction Following Error (The "Rebellious Student"): The Model knows the facts and the logic, but it ignores your specific rules.
- Analogy: You tell the Model, "Write a poem about cats, but do not use the letter 'e'." The Model writes a great poem about cats, but it uses the letter 'e' fifty times. It failed to follow the rules, even though the content was fine.

2. The "PRISM" Lens

The name PRISM is a perfect metaphor. A prism takes a beam of white light and splits it into a rainbow of distinct colors.

Old Benchmarks: Were like a black-and-white camera. They just took a picture of the final answer and said, "Pass" or "Fail." They couldn't tell why it failed.
PRISM: Is like a prism. It takes the Model's messy output and splits it apart so we can see exactly which "color" (Knowledge, Reasoning, or Instructions) is causing the problem.

3. The Big Discovery: The "Trade-Off" Trap

The researchers tested 24 different AI models (both free ones and expensive ones) using this new lens. They found something surprising: You can't fix everything at once.

Think of the Model's brain like a muscle.

If you train it to be super strict about following rules (like a robot that never breaks a command), it might get so focused on the rules that it stops thinking creatively or logically.
If you train it to be a genius at math and logic, it might start ignoring your instructions because it's so busy solving the problem.

The paper calls this a trade-off. Fixing one type of hallucination often makes another type worse. It's like tuning a car: if you tighten the suspension for a smoother ride, the car might handle corners worse.

4. Why This Matters

Before PRISM, developers were like doctors guessing which medicine to give a patient without knowing the diagnosis. They might try to "fix" the Model's memory, only to accidentally break its ability to follow instructions.

With PRISM, we can finally say:

"This Model is great at following rules, but it forgets facts." -> We need to feed it more data.
"This Model knows facts but can't do math." -> We need to train it on logic puzzles.
"This Model is smart but ignores instructions." -> We need to teach it to listen better.

The Bottom Line

PRISM is a new, super-detailed report card for AI. It stops us from just saying "AI is hallucinating" and starts telling us exactly where the AI is hallucinating. This helps engineers build better, safer, and more trustworthy AI assistants that don't just sound smart, but actually are smart and reliable.

1. Problem Statement

As Large Language Models (LLMs) are deployed in high-risk domains (e.g., medical, legal, scientific), the issue of hallucination (generating factually inconsistent content) has become a critical safety concern. However, existing evaluation benchmarks suffer from two fundamental limitations:

Mixed Queries: Current benchmarks (e.g., TruthfulQA, HaluEval) often mix different types of errors (knowledge gaps, reasoning flaws, instruction violations) within single queries. This makes it impossible to isolate the specific cognitive stage where the model failed.
Posterior Evaluation: Most methods evaluate only the final output. They lack "diagnostic" capabilities to determine why a failure occurred (e.g., was it a lack of memory, a logical error, or an inability to follow constraints?).
Lack of Trade-off Analysis: Without granular diagnosis, it is difficult to understand how mitigation strategies (like instruction tuning) might improve one capability while degrading another (e.g., improving formatting but hurting reasoning).

2. Methodology: The PRISM Framework

The authors propose PRISM, a controlled benchmark designed to disentangle hallucinations into four independent dimensions based on three stages of the LLM generation pipeline: Memory, Instruction, and Reasoning.

A. Four Failure Dimensions

PRISM categorizes hallucinations into four orthogonal dimensions:

Knowledge Error (KE): The model possesses the knowledge but retrieves or represents it incorrectly (e.g., factual distortion, intra-memory conflict, entity confusion).
Knowledge Missing (KM): The model lacks the necessary parametric knowledge to answer (e.g., missing domain-specific, fictional, timely, or non-public knowledge).
Reasoning Error (RE): The model has the facts but fails to combine them logically (e.g., logical fallacies, procedural failures, information integration errors, mathematical reasoning).
Instruction Following Error (IFE): The model has the knowledge and reasoning but violates explicit constraints (e.g., format, length, language, complex cognitive loads).

B. Benchmark Construction

Data Source: PRISM contains 9,448 instances across 65 sub-tasks.
Orthogonality Principle: Data subsets are strictly designed to test only one failure mode at a time to prevent confounding variables.
- KE/KM Sources: Wikipedia, Baidu Baike, OOD temporal news (2025), fictional entities, and private domain data.
- RE Sources: Self-contained reasoning tasks (IMO, code generation) where premises are embedded in the context.
- IFE Sources: High-constraint synthetic data involving negative semantics, strict JSON formats, and multi-dimensional constraints.
Construction Pipeline: A three-stage process involving:
1. Data Collection & Cleaning: Noise removal and standardization.
2. Multi-Agent Construction: Four specialized agents (Schema Normalizer, Evidence Retriever, Type Classifier, Quality Scoring) ensure data is grounded, correctly labeled, and high-quality.
3. Human Selection: Domain experts review instances for clarity, relevance, and coherence.

C. Evaluation Protocol

Models Evaluated: 24 mainstream LLMs (both proprietary and open-source), ranging from 8B to 685B parameters (e.g., GPT-5, Gemini-3, Claude-Opus, DeepSeek-R1, Llama-3).
Metrics:
- Accuracy for closed-ended tasks.
- LLM-Eval (0-5 scale) for open-ended tasks.
- Hallucination Rate ( $H$ ): $100 - S$ (where $S$ is the normalized score).
- H-Score: The macro-averaged hallucination rate across all four dimensions.

3. Key Contributions

Diagnostic Framework: PRISM is the first benchmark to systematically decouple hallucinations into KE, KM, RE, and IFE, enabling stage-aware diagnosis.
Comprehensive Evaluation: It provides the first large-scale evaluation of 24 models across 65 sub-tasks, revealing specific weaknesses in different model families.
Trade-off Discovery: The study empirically demonstrates that mitigation strategies often involve inherent trade-offs. For example, strategies that improve instruction following may degrade reasoning capabilities, and knowledge injection can cause catastrophic forgetting.
Mechanism Analysis: The paper analyzes internal mechanisms (e.g., attention maps) to explain why models fail, such as the dominance of parametric priors in KE or misleading context in KM.

4. Key Results

Performance Rankings: Proprietary models generally outperform open-source models. Claude-Opus-4.5, Gemini-3-Pro, and Gemini-3-Flash achieved the lowest overall hallucination rates (H-Scores of ~13.9% - 15.5%).
Dimensional Disparities:
- Reasoning (RE) and Knowledge Error (KE) show the largest performance gaps between models.
- Knowledge Missing (KM) errors are relatively low for most models, suggesting that current models are better at admitting ignorance than hallucinating facts when knowledge is missing.
- Open-Source vs. Proprietary: Open-source models approach proprietary performance in KM but lag significantly in RE and IFE, particularly on complex multi-step tasks.
Mitigation Trade-offs (RQ2):
- Instruction Tuning: Improves instruction following and reduces knowledge conflicts but often degrades reasoning capabilities.
- Reasoning Tuning (SFT): Significantly improves reasoning but can worsen knowledge retrieval and instruction adherence.
- In-Context Learning (ICL): Increasing shots (0-shot to 3-shot) yields marginal gains, primarily helping with format rather than root causes.
Internal Mechanisms (RQ3):
- KE: Models exhibit "diffuse attention," failing to focus on specific evidence tokens that contradict their parametric priors.
- KM: Models often attend overly to misleading context tokens when internal knowledge is absent, prioritizing context consistency over factual accuracy.
- Computational Cost: Correctly refusing an instruction (IFE) requires more active neurons and GFLOPs than generating a hallucinated answer, suggesting hallucination is computationally "cheaper" for the model.

5. Significance and Future Work

Trustworthy AI: PRISM provides a quantitative framework for understanding the specific mechanisms behind hallucinations, moving beyond "black box" error rates to actionable diagnostics.
Optimization Guidance: By revealing the trade-offs between dimensions, PRISM guides the development of balanced mitigation strategies rather than those that optimize one metric at the expense of others.
Limitations: The current benchmark focuses on text modality and five high-resource languages. Future work aims to extend PRISM to multimodal settings, incorporate dynamic temporal updates, and address low-resource languages.

In conclusion, PRISM shifts the paradigm of hallucination evaluation from outcome-based scoring to process-based diagnosis, offering a critical tool for researchers and developers to build more reliable and trustworthy LLMs.