Beyond Final Answers: CRYSTAL Benchmark for Transparent Multimodal Reasoning Evaluation

This paper introduces CRYSTAL, a transparent multimodal reasoning benchmark that evaluates models via verifiable intermediate steps using novel metrics like Match F1 and Ordered Match F1, while proposing the Causal Process Reward (CPR) and CPR-Curriculum training strategies to significantly improve reasoning quality without manual step annotation.

The Gist

Imagine you are taking a math test in school. The teacher asks you to solve a problem: "If a train leaves Station A at 60 mph and another leaves Station B at 40 mph, when do they meet?"

In the old way of testing AI (the "Answer-Only" method), the teacher only looks at the final number you write down.

• Student A writes down the correct answer, 2 hours, but they actually just guessed, or they wrote down the wrong steps like "50 + 50 = 100" and somehow got the right number by luck. The teacher gives them an A.
• Student B writes down the correct answer, 2 hours, and shows their work: "Distance = Speed × Time... here is the algebra... therefore 2 hours." The teacher also gives them an A.

The problem? Student A is a liar. They don't actually understand math; they just got lucky. But because the teacher only checked the final box, they can't tell the difference between a genius and a guesser.

This is exactly what the paper "CRYSTAL" is trying to fix.

🧊 What is CRYSTAL?

CRYSTAL stands for Clear Reasoning via Yielded Steps, Traceability, and Logic.

Think of CRYSTAL not as a test, but as a transparent glass box around the AI's brain. Instead of just looking at the final answer, CRYSTAL forces the AI to show its work, step-by-step, like a detective writing down every clue they found before solving the case.

🔍 The "Lucky Guess" Problem

The paper shows a funny example: An AI looks at a picture of three video game consoles and is asked, "Which one is the smallest?"

• The AI says: "The middle one." (Correct answer!)
• The AI's reasoning: "The middle one is the largest of all, so I will pick it as the smallest."

In the old system, the AI gets a perfect score because the answer was right. In the CRYSTAL system, the AI gets a failing grade because its logic is backwards. It's like a chef who serves you a delicious cake but tells you, "I burned the flour and added salt," yet the cake tastes sweet. You know something is wrong with the process, even if the result looks good.

🏆 How CRYSTAL Grades (The Two New Metrics)

CRYSTAL uses two special rulers to measure the AI, not just the answer:

1. Match F1 (The "Did you say it?" Ruler):
   Imagine the "perfect" solution is a list of 10 specific clues.

   • If the AI lists 10 clues but 5 of them are made up (hallucinations), it gets a low score.
   • If the AI lists 2 clues that are perfect but misses the other 8, it also gets a low score.
   • This measures if the AI actually found the right evidence.

2. Ordered Match F1 (The "Did you say it in the right order?" Ruler):
   Imagine you are giving someone directions to a party.

   • Wrong Order: "Turn left at the bakery, then go to the park, then turn right at the library." (This is confusing and wrong).
   • Right Order: "Go to the library, turn right, then go to the park, then turn left at the bakery."
   • CRYSTAL checks if the AI's steps make logical sense in the order they were written. The paper found that even the smartest AIs often get the steps right but put them in the wrong order, like a jumbled puzzle.

🎓 The "Cherry-Picking" Discovery

The researchers tested 20 different AI models (including the super-smart ones from big tech companies). They found a shocking habit called "Cherry-Picking."

• What it is: The AI looks at a problem, finds one tiny clue that helps it guess the answer, ignores the other 90% of the clues, and just says the answer.
• The Metaphor: It's like a student taking a test who only reads the first sentence of the question, guesses the answer, and ignores the rest of the paragraph. They get the right answer 50% of the time, but they aren't actually "thinking."
• The Result: The paper found that almost every AI does this. They are great at guessing the final answer but terrible at showing the full path to get there.

🚀 The Solution: CPR (Causal Process Reward)

So, how do we fix an AI that loves to guess? The authors invented a new training method called CPR.

• Old Training: "If you get the answer right, you get a cookie. If your reasoning is good, you get a little extra cookie."
  • Result: The AI learns to just guess the answer to get the big cookie and ignores the reasoning.
• CPR Training: "You only get a cookie if BOTH the answer is right AND the reasoning is good. If you guess right but have bad reasoning, you get NO cookie."
  • Result: The AI is forced to learn how to think properly because it can't get the reward any other way.

They also added a "Curriculum" (like school grades). They started the AI on easy problems with short reasoning chains and slowly made the problems harder. This helped the AI learn to think step-by-step without getting overwhelmed.

💡 The Big Takeaway

This paper is a wake-up call. Just because an AI gives you the right answer doesn't mean it understands the world. It might just be a very good guesser.

CRYSTAL is a new tool that forces AI to show its homework. By using this tool and the new CPR training method, the researchers were able to teach an AI to not just guess the answer, but to actually understand the logic behind it, improving its reasoning skills by 32% without needing humans to write out every single step for it.

In short: Stop asking AI "What is the answer?" and start asking "How did you get there?"

Technical Summary

1. Problem Statement

Current Multimodal Large Language Models (MLLMs) are evaluated primarily on final answer accuracy (e.g., "Is the answer correct?"). This approach suffers from a critical flaw: it cannot distinguish between genuine reasoning and shortcuts (e.g., lucky guesses, cherry-picking, or hallucinated logic that coincidentally leads to the right answer).

• The "Lucky Guess" Problem: A model might output a correct answer while its intermediate reasoning steps are contradictory, hallucinated, or logically flawed. Traditional benchmarks award full credit, masking these failures.
• Incentive Misalignment: Answer-centric evaluation structurally incentivizes models to guess confidently rather than signal uncertainty or provide transparent, verifiable reasoning chains.
• Lack of Granularity: Existing process evaluation methods often lack machine-verifiable intermediate checkpoints or fail to decouple visual perception from symbolic reasoning.

2. Methodology: The CRYSTAL Framework

The authors introduce CRYSTAL (Clear Reasoning via Yielded Steps, Traceability and Logic), a diagnostic benchmark and evaluation framework designed to assess reasoning step-by-step.

A. Dataset Construction (Delphi-Inspired Pipeline)

To create high-quality reference reasoning steps without manual annotation for every instance, the authors developed a multi-agent pipeline:

1. Independent Generation: Four diverse open-source MLLMs (Qwen2.5-VL, InternVL3, Gemma3, Llama-4) independently generate reasoning trajectories for 6,372 questions derived from existing benchmarks (MathVista, ScienceQA, RealWorldQA, etc.).
2. Semantic Clustering: Steps are embedded and clustered based on semantic similarity. Redundant or paraphrased steps are merged to form a consensus set.
3. Automated Validation: A fifth MLLM validates the logical soundness, visual grounding, and consistency of the clustered steps.
4. Human Quality Gate: Human annotators verify the final steps. Less than 5% of examples require re-iteration.

• Scale: 6,372 instances with an average of 11.6 reasoning steps per question.

B. Evaluation Metrics

CRYSTAL introduces two novel metrics to evaluate the generated reasoning steps against the reference steps:

1. Match F1: Measures step-level precision and recall using semantic similarity (cosine similarity with all-distilroberta-v1).
   • Precision: Fraction of predicted steps that match reference steps.
   • Recall: Fraction of reference steps covered by the prediction.
   • Significance: High precision but low recall indicates "cherry-picking" (only stating safe, correct facts while omitting necessary logic).
2. Ordered Match F1: Extends Match F1 by penalizing disordered reasoning chains using the Longest Increasing Subsequence (LIS) ratio. This ensures the model not only finds the right steps but arranges them in a logically coherent sequence.

C. Training Strategy: Causal Process Reward (CPR)

To improve model reasoning, the authors propose a new reinforcement learning reward strategy:

• Causal Process Reward (CPR): Unlike additive rewards (Accuracy + Reasoning), CPR uses a multiplicative coupling. The model receives a full reward only if the answer is correct AND the reasoning steps align with the reference.
  • Formula: $R_{CPR} = (a_w + s_w \cdot F1_{step})$ if correct; otherwise heavily discounted.
  • This prevents the model from maximizing reward by guessing the answer while ignoring reasoning quality.
• CPR-Curriculum: A two-phase training strategy.
  • Phase 1: Train on format and answer accuracy only (no reasoning signal) to stabilize generation.
  • Phase 2: Introduce the full CPR reward with progressive difficulty (starting with fewer steps, increasing to complex chains).

3. Key Contributions

1. CRYSTAL Benchmark: A diagnostic dataset of 6,372 instances with verifiable intermediate reasoning steps, enabling fine-grained evaluation of multimodal reasoning.
2. Novel Metrics: Match F1 and Ordered Match F1, which quantify reasoning quality, coverage, and logical ordering, exposing failures invisible to accuracy metrics.
3. Training Paradigm: The Causal Process Reward (CPR) and CPR-Curriculum, which successfully train models to generate faithful reasoning chains without manual step annotation during deployment.
4. Empirical Findings: A comprehensive evaluation of 20 MLLMs (including frontier commercial models like GPT-5) revealing systematic reasoning flaws.

4. Key Results & Findings

The evaluation of 20 MLLMs (16 open-source, 4 commercial) yielded several critical insights:

• Universal Cherry-Picking: 19 out of 20 models exhibit "cherry-picking" behavior, where Precision significantly exceeds Recall (ratios from 1.2x to 7.2x). Models generate a few high-confidence, correct steps but omit the bulk of the necessary reasoning chain. Even GPT-5 (57.99% accuracy) only recovers 47.9% of reference steps.
• Divergence of Accuracy and Reasoning: High accuracy does not correlate with high reasoning quality.
  • Example: GPT-5 leads in accuracy (57.99%) but ranks 8th in Match F1.
  • Example: GPT-5-mini has the highest Match F1 (0.773) but slightly lower accuracy (55.59%).
  • Scaling Issues: Larger models do not always improve reasoning. Qwen3-VL-32B has better reasoning (F1 0.718) but lower accuracy than Qwen3-VL-8B, indicating a trade-off between thoroughness and answer extraction.
• Disordered Reasoning: No competitive model preserves more than 60% of matched steps in the correct logical order (Ordered Match F1). Models retrieve relevant facts but fail to organize them into a coherent chain.
• Training Success:
  • Additive reward strategies (Composite) fail, often collapsing or ignoring reasoning.
  • CPR-Curriculum achieves a +32% improvement in Match F1 (from 0.480 to 0.633) and a +7.67% increase in accuracy on Qwen2.5-VL-3B.
  • The method transfers across architectures (tested on InternVL3.5-4B), nearly tripling recall (0.325 → 0.811).

5. Significance

• Transparency: CRYSTAL shifts the paradigm from "black-box" answer evaluation to "white-box" process evaluation, revealing that current SOTA models often rely on shortcuts rather than genuine understanding.
• Diagnostic Power: The metrics allow researchers to pinpoint exactly where models fail (e.g., perception errors vs. logical ordering vs. cherry-picking).
• Training Guidance: The CPR framework demonstrates that it is possible to train models to produce faithful, step-by-step reasoning without expensive human annotation of every step, simply by coupling answer correctness with step-level alignment.
• Future Direction: The paper highlights that current scaling laws may not improve reasoning quality linearly and suggests that future MLLMs must be trained with process-level constraints to achieve trustworthy, transparent reasoning.