💻 computer science

CoRE: A Fine-Grained Code Reasoning Benchmark Beyond Output Prediction

The paper introduces CoRE, a fine-grained code reasoning benchmark that evaluates large language models on implementation invariance and process transparency, revealing that current models often lack robustness across equivalent code implementations and rely on superficial execution rather than genuine intermediate state reasoning.

Original authors: Jun Gao, Yun Peng, Qian Qiao, Changhai Zhou, Yuhua Zhou, Shiyang Zhang, Shichao Weng, Zhenchang Xing, Xiaoxue Ren

Published 2026-04-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jun Gao, Yun Peng, Qian Qiao, Changhai Zhou, Yuhua Zhou, Shiyang Zhang, Shichao Weng, Zhenchang Xing, Xiaoxue Ren

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Are AI Coders Actually "Thinking"?

Imagine you hire a brilliant student to solve a math problem. They write down the final answer, and it's correct. You give them a gold star.

But then, you ask them to explain how they got there. They start making up numbers, mixing up steps, and getting confused about the middle of the process. Yet, somehow, they still landed on the right final number.

The Problem: Current tests for AI (Large Language Models or LLMs) are like that gold star. They only check the final answer. If the AI gets the right output, we assume it "understood" the code. The authors of this paper argue that this is a trap. The AI might be guessing or using shortcuts (heuristics) rather than actually simulating how the code runs in its "mind."

The Solution: CoRE (The "Honesty Test" for Code)

The authors created a new benchmark called CoRE (Code Reasoning). Instead of just asking, "What is the answer?", CoRE asks two much harder questions:

Does it matter how you ask? (Implementation Invariance)
Do you know what happened in the middle? (Process Transparency)

Analogy 1: The "Same Cake, Different Recipes" (Implementation Invariance)

Imagine you ask a chef to bake a chocolate cake.

Standard Test: You give them one specific recipe (Recipe A). They bake it. It tastes good. Pass.
CoRE Test: You give them four different recipes for the exact same chocolate cake.
- Recipe 1: Uses cocoa powder.
- Recipe 2: Uses melted chocolate bars.
- Recipe 3: Uses a different mixing bowl order.
- Recipe 4: Uses a different oven temperature.

If the chef is truly a master, they should be able to bake a perfect cake regardless of which recipe they use.
The Paper's Finding: The AI models failed this. They were great at baking the cake when using a recipe that looked like the one they usually write (their own "style"). But when given a recipe written by a different AI (a different "style"), they often messed up, even though the math was identical. They were "style-obsessed" rather than logic-obsessed.

Analogy 2: The "Movie Director" vs. The "Spoiler" (Process Transparency)

Imagine you are watching a movie.

Standard Test: The AI is asked, "Who wins the movie?" It says, "The hero wins." Correct!
CoRE Test: The AI is asked, "At minute 45, when the hero is hiding in the closet, what is the villain holding? And at minute 60, what is the hero's score?"

This is checking the intermediate states.
The Paper's Finding: The AI models often got the "Who wins?" question right, but they hallucinated the details in the middle. They guessed the ending based on patterns, but they didn't actually "watch" the movie step-by-step. They were superficially executing the code without truly understanding the journey.

How They Built CoRE

To make this test fair and hard, the researchers did two main things:

Generated Many Versions: They used different AI models to write code for the same 60 problems. This created a library of 255 different code versions. Some were long and messy; some were short and clever. But they all did the exact same thing.
Created "Pop Quiz" Questions: For every piece of code, they generated questions about the middle of the process.
- Arithmetic: "What is the exact number in this variable right now?"
- Logic: "Is this condition true or false?"
- State: "How many times has this loop run so far?"
- Boundary: "What happens right when the loop stops?"

What They Found (The Results)

They tested 8 of the smartest AI models available (like GPT-5, Claude, DeepSeek, etc.) using this new benchmark. The results were surprising:

The "Style Bias" Gap: The models were much better at understanding code written by their own "family" (e.g., OpenAI models understood OpenAI code best) than code written by others. This suggests they are memorizing styles, not learning universal logic.
The "Superficial Execution" Gap: Many models got the final answer right but failed the "pop quiz" about the middle steps. They were guessing the destination without knowing the path.
The "RCS" Score: The authors created a new score called Reasoning Consistency Score (RCS). This score punishes models that get the right answer for the wrong reasons. If you get the answer right but fail the middle steps, your score drops to zero.

The Conclusion

The paper concludes that getting the right answer isn't enough. Just because an AI can predict the final output of a code snippet doesn't mean it understands how the code works.

Current AI models are like actors who memorize the final line of a play but forget the dialogue in between. CoRE is the new test that forces them to show their work, proving that they are actually "thinking" through the code, not just guessing the ending.

In short: CoRE exposes that today's smartest code-writing AIs are often "faking it" by relying on patterns and styles rather than genuine, step-by-step reasoning.

1. Problem Statement

Despite the rapid advancement of Large Language Models (LLMs) in code generation, it remains unclear whether these models possess a genuine understanding of code execution logic or merely rely on superficial heuristics to predict correct outputs. Existing benchmarks (e.g., CruxEval, LiveCodeBench-O, REval) suffer from two critical flaws:

Lack of Implementation Invariance: They typically evaluate models against a single "canonical" code implementation. This fails to test if a model can reason consistently across functionally equivalent but structurally or lexically distinct code variations.
Lack of Process Transparency: They focus almost exclusively on final output correctness. Models can often produce the correct final answer while hallucinating intermediate execution states, indicating a lack of true step-by-step reasoning.

The authors argue that output-only evaluations are insufficient for assessing robust code reasoning and introduce CoRE to address these gaps.

2. Methodology: The CoRE Benchmark

CoRE is constructed from 60 coding problems selected from HumanEval and LiveCodeBench, designed to evaluate two core dimensions: Implementation Invariance and Process Transparency.

A. Construction Pipeline

Diverse Code Implementations (Stage I):
- The authors used seven diverse LLMs to generate multiple solutions for each problem.
- Cleaning Process: Solutions underwent execution validation to ensure functional equivalence. To ensure diversity, they filtered for high Cyclomatic Complexity and low 1-gram Jaccard Similarity (lexical diversity) between implementations of the same problem.
- Result: 255 unique code implementations across 60 problems (average 4.3 implementations per problem).
Intermediate State Probes (Stage II):
- Using Python execution tracing, the authors captured ground-truth runtime values for all variables.
- They formulated intermediate probing questions targeting four reasoning dimensions:
  - Arithmetic: Numerical precision in complex operations.
  - Logic: Resolution of nested boolean conditions and control flow.
  - State: Tracking variable history across deep execution paths.
  - Boundary: Critical transition points (e.g., loop terminations).
- Result: 243 intermediate probes (average 4.1 per problem), with each probe covering an average of 2.2 reasoning dimensions.

B. Evaluation Protocol & Metrics

The authors propose a holistic evaluation protocol that moves beyond simple accuracy:

Strict Output Consistency ( $I$ ): A binary indicator (1 or 0) where the model must predict the correct output for every diverse implementation of a problem.
Process Fidelity Weight ( $W_s$ ): The ratio of correctly answered intermediate state probes.
Reasoning Consistency Score (RCS): The primary metric defined as $RCS = I \times W_s$ $R C S = I \times W_{s}$ .
- This formulation ensures that if a model fails implementation invariance (i.e., gets one variant wrong) or relies on superficial heuristics (low $W_s$ ), the final score is heavily penalized or nullified.

3. Key Contributions

CoRE Benchmark: A rigorous benchmark comprising 1,978 samples that jointly evaluates implementation invariance and process transparency, featuring higher complexity (avg. cyclomatic complexity of 5.0) and diversity than existing benchmarks.
Identification of Fundamental Limitations: The study reveals two critical failure modes in frontier LLMs:
- Robustness Gap (Style Overfitting): Models perform significantly better on code generated by their own family or familiar styles (e.g., OpenAI models on OpenAI code) compared to functionally equivalent code from other styles.
- Superficial Execution: Models frequently arrive at correct final outputs while hallucinating intermediate states, proving they rely on input-output mapping rather than genuine execution simulation.
Holistic Evaluation Protocol: The introduction of the Reasoning Consistency Score (RCS) provides a method to distinguish genuine reasoning from heuristic shortcuts.

4. Experimental Results

The authors evaluated eight frontier LLMs (including GPT-5, o3, Claude-4.5, DeepSeek-V3.2, and Qwen-3) using various prompting strategies (IO, Chain-of-Thought, Chain-of-Code, RHDA).

Performance Gap: There is a massive disconnect between final output accuracy and intermediate reasoning. For example, GPT-5 achieved an 84.62% strict output accuracy ( $I$ ) but only 18.96% process fidelity ( $W_s$ ) under standard prompting.
Style Bias: Heatmap analysis (Fig. 5) shows a strong diagonal dominance, where models score highest on code generated by their own family. This confirms stylistic overfitting.
Impact of Prompting: While Chain-of-Thought (CoT) and Chain-of-Code (CoC) improved process fidelity, they did not fully eliminate the robustness gap or superficial execution.
Dimensional Weaknesses: Models struggled most with Arithmetic (numerical precision) and State (maintaining variable history), with average process fidelity scores often below 50% in the IO setting.
RCS Effectiveness: The RCS metric effectively filtered out models that appeared competent based on output alone but failed on reasoning consistency.

5. Significance

The paper fundamentally shifts the paradigm of code reasoning evaluation. It demonstrates that correct output does not imply correct reasoning. By exposing the fragility of current LLMs—specifically their susceptibility to stylistic bias and their inability to track intermediate states—CoRE provides a necessary standard for developing models that truly understand code execution. The findings suggest that future LLM training and evaluation must prioritize faithful step-by-step reasoning and robustness across diverse implementations rather than just optimizing for final output accuracy.