How Robustly do LLMs Understand Execution Semantics?

1. Problem Statement

While Large Language Models (LLMs) have demonstrated impressive proficiency in software engineering tasks (e.g., code generation, summarization), it remains unclear whether this performance stems from a genuine understanding of code semantics or merely sophisticated pattern matching. Current benchmarks often test static knowledge or multiple-choice selection, which may not reflect a model's ability to robustly predict program behavior under varying conditions.

The core problem addressed is the lack of robustness in LLM code understanding. Specifically, the authors investigate:

• Whether models can consistently predict program outputs when inputs are perturbed (e.g., type-aware mutations).
• Whether models maintain performance when code is transformed via meaning-preserving transformations (MPTs).
• How well models handle runtime exceptions, which are critical for debugging but often ignored in standard benchmarks.
• Whether the complexity of execution paths (number of control flow decisions) degrades reasoning capabilities.

2. Methodology

The study utilizes the CruxEval benchmark, which requires models to predict exact outputs for given Python code and inputs. The authors extend this benchmark through three experimental dimensions corresponding to their Research Questions (RQs):

A. Input Perturbation (RQ1)

• Technique: Applied type-aware mutation to generate a dense neighborhood of local inputs for each program.
• Exception Handling: Inputs that caused runtime exceptions (e.g., IndexError, TypeError, TimeoutError) were recorded in a new dataset, CruxEval_exc.
• Metrics:
  • Accuracy ($C_o$, $C_{\tilde{o}}$): Performance on original vs. perturbed inputs.
  • Robust Drop ($R\Delta$): The performance degradation ($Accuracy_{perturbed} - Accuracy_{original}$).
  • Program-level Strict Robustness (PSR): The fraction of programs where the model predicts all inputs correctly (a stricter measure of true understanding).
  • Exception Prediction: Evaluated via "Strict Match" (exact exception type/message) and "Any Match" (predicting some exception occurred).

B. Program Transformation (RQ2)

• Technique: Applied Meaning-Preserving Transformations (MPTs) (e.g., operand swapping, block swapping, dead code insertion) and Variable Renaming to the source code.
• Goal: To test if models rely on syntactic patterns rather than semantic logic.

C. Execution Trace Complexity (RQ3)

• Technique: Used dynamic analysis (tracing) to count the number of decisions (control flow branches, loop iterations, conditions) encountered during execution.
• Goal: To determine if reasoning accuracy correlates with the complexity of the execution path.

D. Model Selection

The study evaluated 14 models, including:

• Open-weights: Qwen2.5 (Math, Base, Instruct), DeepSeek R1 Distilled (various sizes), Llama 3.1, Nemotron.
• Frontier/Proprietary: GPT-5 Nano, GPT-5.2, and Gemini 3 Pro.
• Configuration: Evaluated both base and instruction-tuned variants, with a focus on reasoning capabilities.

3. Key Contributions

1. Robustness Metrics for Code: Introduced Program-level Strict Robustness (PSR) and Robust Drop ($R\Delta$) to quantify consistency beyond average accuracy.
2. Extension of CruxEval: Created CruxEval_exc, a dataset specifically targeting runtime exceptions, and analyzed the "sycophantic" tendency of models to ignore errors when not explicitly prompted.
3. Prompt Engineering Interventions: Demonstrated that modifying prompts to explicitly request type tracking and exception analysis significantly improves exception prediction, though it may negatively impact performance on non-exceptional inputs for some models.
4. Comprehensive Benchmarking: Provided a granular analysis of how different model architectures (reasoning vs. non-reasoning, open vs. closed) respond to semantic perturbations.

4. Key Results

RQ1: Input Perturbation & Exceptions

• Frontier Model Brittleness: While GPT-5.2 achieved near-perfect accuracy (99.12%) on the original CruxEval, it suffered a massive Robust Drop of -19.52% on perturbed inputs. Its PSR dropped to 56%, indicating it fails to consistently understand the same program across different inputs.
• Surprising Robustness: GPT-5 Nano (a smaller, likely quantized/distilled version) outperformed GPT-5.2 on perturbed inputs, suggesting quantization may act as a regularizer, improving generalization.
• Exception Prediction Failure: On the original prompt, frontier models performed poorly on exception prediction (GPT-5.2: ~15% accuracy). This was attributed to the prompt not explicitly asking for exception handling, leading models to "hallucinate" valid outputs instead of errors.
• Prompt Fixes: Adding explicit instructions to track types and exceptions improved GPT-5.2's exception prediction from ~15% to ~81%. However, this same prompt degraded GPT-5.2's performance on non-exceptional inputs (dropping from 99% to ~76%), suggesting a trade-off where the model becomes overly sensitive to error conditions.

RQ2: Program Transformation

• Syntactic Sensitivity: GPT-5.2 showed significant sensitivity to code transformations (MPTs and variable renaming), dropping from ~99% to ~76%.
• Gemini 3 Pro Resilience: In contrast, Gemini 3 Pro maintained high robustness, dropping only slightly (99% $\to$ 95-98%) under similar transformations.
• Open-Source Performance: Reasoning-distilled models (e.g., DeepSeek R1) showed significant gains over their base counterparts but still exhibited some degradation under perturbation.

RQ3: Control Flow Decisions

• Decision Complexity: There is a negative correlation between the number of decisions in an execution path and prediction accuracy. As the number of decisions increases (e.g., nested loops, complex if-else chains), accuracy generally declines.
• Gemini 3 Pro Outlier: Unlike other models, Gemini 3 Pro showed a positive or neutral correlation with decision count, suggesting a superior ability to reason through complex control flows.

5. Significance and Implications

• Limitations of "Understanding": The study provides strong evidence that even state-of-the-art frontier models (like GPT-5.2) lack robust semantic understanding. Their high accuracy on standard benchmarks is fragile and collapses under minor perturbations or when asked to reason about errors.
• Evaluation Methodology: The paper argues that perturbation testing (input mutation and code transformation) is essential for evaluating code models. Standard benchmarks are insufficient as they may reward memorization or pattern matching over true execution semantics.
• Debugging Reliability: The findings suggest that developers cannot fully rely on current LLMs for debugging tasks involving exceptions or complex control flows without explicit prompting, and even then, performance trade-offs exist.
• Future Directions: The work highlights the need for better prompt engineering strategies and model architectures that can balance robustness across normal execution, edge cases, and exception handling without sacrificing baseline performance.

In conclusion, while LLMs are powerful tools, their "understanding" of code execution semantics is currently brittle, particularly for frontier models that excel on static benchmarks but fail under dynamic, perturbed, or exception-heavy scenarios.