Evaluating LLMs Code Reasoning Under Real-World Context

1. Problem Statement

Current benchmarks for evaluating Large Language Models' (LLMs) code reasoning capabilities suffer from significant limitations regarding real-world applicability:

• Simplistic Data Structures: Existing benchmarks (e.g., CRUXEval, HumanEval, ClassEval) predominantly rely on primitive data types (integers, strings) or LLM-generated snippets. They fail to capture the complexity of compound and custom objects found in production code.
• Lack of Contextual Dependencies: Most datasets ignore the intricate inter- and intra-procedural dependencies, third-party API usages, and class hierarchies typical of real-world repositories.
• Evaluation Gap: While state-of-the-art LLMs perform exceptionally well on these simplified benchmarks, their ability to generalize to complex, real-world software engineering tasks remains unverified and likely overestimated.
• Serialization Challenges: Real-world projects often involve custom objects that lack __str__() implementations, making standard text-based serialization (printing) ineffective for capturing actual runtime values.

2. Methodology: R2Eval

To address these gaps, the author introduces R2Eval (Realistic Code Reasoning Evaluation), a benchmark designed to assess code reasoning under realistic conditions.

• Data Source: The benchmark consists of 135 reasoning problems extracted from ten widely used, mature Python projects: scikit-learn, django, requests, seaborn, sphinx, pytest, astropy, xarray, matplotlib, and sympy.
• Core Innovation: Custom Type Serialization:
  • Unlike prior work that discards non-primitive inputs, R2Eval employs a pipeline combining static and dynamic program analysis.
  • It iteratively decomposes complex, custom data types (e.g., class instances) into their constituent primitive or core compound types.
  • These structures are serialized into JSON formats, allowing LLMs to "see" the actual state of complex objects (e.g., internal attributes of a WCS object) rather than just memory addresses.
• Evaluation Mechanism:
  • Task Format: Each problem is a triplet $\{P, I, O\}$, where $P$ is the code (including dependencies and class context), and $I/O$ are the serialized inputs/outputs.
  • Prediction Tasks: LLMs are tasked with Input Prediction (inferring inputs given code and outputs) and Output Prediction (inferring outputs given code and inputs).
  • Robust Verification: To avoid false negatives caused by textual comparison of complex objects, R2Eval deserializes the LLM's predicted JSON back into Python objects and executes runtime tests to verify correctness.

3. Key Contributions

1. R2Eval Benchmark: A novel dataset of 135 problems derived from real-world open-source projects, preserving the structural and data complexity of production code.
2. Serialization Pipeline: A technical solution for automatically serializing and deserializing custom Python objects into JSON, enabling LLMs to reason over non-primitive types without losing semantic information.
3. Runtime Verification Framework: A scalable method for evaluating LLM predictions by reconstructing objects and running unit tests, ensuring accurate assessment beyond string matching.
4. Empirical Evidence of Generalization Failure: The study provides concrete data showing that high performance on existing benchmarks does not correlate with performance on realistic code.

4. Experimental Results

The author evaluated six prominent LLMs (including o4-mini, Gemini-2.5-Pro, DeepSeek-R1, GPT-4.1, and DeepSeek-Coder) on both R2Eval and the standard CRUXEval benchmark.

• Significant Performance Drop:
  • Moving from CRUXEval to R2Eval, average performance dropped by 64.32% for Input Prediction and 52.22% for Output Prediction.
  • Example: o4-mini dropped from 92.59% (CRUXEval Input) to 20.00% (R2Eval Input).
• Reasoning vs. Non-Reasoning Models: "Reasoning" LLMs (e.g., o4-mini, DeepSeek-R1) consistently outperformed non-reasoning models on both benchmarks, but the absolute performance gap remained large on R2Eval.
• Directional Difficulty:
  • Input prediction proved significantly harder than output prediction in real-world contexts.
  • The overlap between models succeeding in both tasks decreased drastically on R2Eval compared to CRUXEval.
  • Hypothesis: Real-world code complexity (dependencies, custom types) makes backward reasoning (Input Prediction) substantially more difficult than forward reasoning (Output Prediction).

5. Significance and Impact

• Re-evaluating LLM Capabilities: The results challenge the "bold claims" regarding LLMs' code reasoning abilities, suggesting that current models are heavily overfitted to simplified, primitive-heavy benchmarks.
• Benchmark Evolution: R2Eval sets a new standard for code evaluation by mandating the inclusion of custom types and real-world dependencies, pushing the community toward more rigorous testing.
• Future Directions: The paper highlights the need for LLMs to better understand object-oriented structures, state management, and complex data flows to be viable for practical software engineering tasks like program repair and translation in production environments.

In conclusion, R2Eval serves as a critical stress test, revealing that while LLMs excel at abstract code puzzles, they currently struggle significantly with the messy, complex reality of actual software projects.