Technical Summary: Are Large Language Models Robust in Understanding Code Against Semantics-Preserving Mutations?

Problem Statement

The widespread adoption of Large Language Models (LLMs) in software development, particularly for tasks like code generation and debugging, has raised critical questions regarding their true understanding of code semantics. While recent studies assess LLMs' ability to predict program outputs, they often focus solely on accuracy, neglecting the underlying reasoning processes. There is a growing concern that models may arrive at correct answers through flawed logic or pattern recognition rather than genuine semantic understanding. This paper investigates whether state-of-the-art LLMs can robustly reason about Python programs or if their performance is merely a result of guessing or superficial syntactic pattern matching. Specifically, it examines if LLMs exhibit stable, semantically grounded reasoning when subjected to semantics-preserving code mutations—transformations that alter syntax without changing runtime behavior.

Methodology

The authors evaluated nine state-of-the-art LLMs (six open-source and three closed-access/proprietary) using two established benchmarks: LIVECODEBENCH and CRUXEVAL. The evaluation framework consisted of two primary components:

1. Expert Reasoning Analysis: A manual, blind expert evaluation was conducted on the LIVECODEBENCH dataset to categorize correct predictions into three distinct categories:

   • Sound Reasoning: The model correctly predicts the output and accurately describes the control and data flow.
   • Flawed Reasoning: The model predicts the correct output but provides an incorrect or logically unsound explanation of the code's execution.
   • Guessing: The model arrives at the correct answer without a coherent reasoning trace.
   • The study also employed an iterative querying strategy, where incorrect predictions were followed by feedback prompts to assess if models could refine their reasoning.

2. Robustness via Semantics-Preserving Mutations: The authors applied five specific code transformations to the benchmarks to test model stability. These mutations preserve program semantics but alter syntax:

   • Variable Renaming: Replacing identifiers with fresh, non-conflicting names.
   • Comparison Expression Mirroring: Swapping operands and applying logically equivalent inverse operators (e.g., a == b to b == a).
   • Swap If-Else Branches: Swapping the if and else blocks while negating the condition.
   • For-to-While Loop Conversion: Rewriting for loops into semantically equivalent while loops.
   • Partial Loop Unrolling: Extracting the final iterations of a loop and executing them sequentially after the loop.

The authors measured the change in prediction accuracy and stability across these mutations to determine if models rely on invariant semantic understanding or surface-level syntactic cues.

Key Contributions

The paper makes the following contributions:

• Expert Evaluation of Reasoning: A manual analysis determining whether correct code predictions are derived from sound logic, flawed reasoning, or mere guessing.
• Robustness Benchmarking: An evaluation of LLM output prediction stability across five distinct semantics-preserving code mutations.
• Empirical Evidence of Fragility: Demonstration that LLMs frequently alter predictions when subjected to benign syntactic changes, indicating a lack of stable, semantically grounded reasoning.
• Research Directions: A discussion on integrating formal methods with machine learning to create semantics-aware code models that are robust against adversarial mutations.

Results

The experiments yielded several critical findings regarding the reasoning capabilities and robustness of current LLMs:

• Flawed Reasoning in Correct Predictions: A significant portion of correct predictions are based on flawed reasoning. For general-purpose models, up to 55% of correct answers were based on flawed reasoning. For models specifically trained for coding, this figure reached 50%.
• Impact of Interactive Querying: Interactive querying (providing feedback on incorrect answers) improved both prediction accuracy and reasoning quality. It increased the rate of sound reasoning by 14% for coding-specialized models and 19% for the general-purpose Llama3.2 model.
• Sensitivity to Syntax: Models frequently changed their predictions in response to semantics-preserving mutations, with performance drops reaching up to 70%.
  • Variable Renaming: Consistently caused substantial changes in predictions across both datasets, suggesting that identifier names strongly influence model behavior.
  • Loop Unrolling: Led to the largest performance drops, highlighting limited structural and semantic generalization.
• Proprietary vs. Open-Source Performance:
  • Accuracy: Proprietary models (GPT-5.2 and Gemini-3) achieved near-perfect predictive accuracy (97.5% and 100% on LIVECODEBENCH, respectively) and the highest quality of sound reasoning in initial attempts.
  • Robustness: Despite high baseline accuracy, these top-tier models exhibited substantial fragility under mutations. For instance, Gemini-3's accuracy dropped by 70.4% under loop unrolling, and GPT-5.2 dropped by 68.1%.
  • Caveats: The authors note that the superior performance of GPT-5.2, Gemini-3, and Qwen3-Coder may be partially attributed to potential data contamination, as these models were released after or around the time the LIVECODEBENCH benchmark was published (2024), whereas most open-source models were released prior.

Significance and Claims

The paper claims that accuracy alone is not a reliable proxy for genuine code understanding. The findings demonstrate that even state-of-the-art models, including proprietary systems with high predictive accuracy, fail to maintain stable predictions under semantics-preserving transformations. This indicates that current LLMs do not yet exhibit true semantic understanding but rather rely on pattern recognition and surface-level syntactic cues.

The authors argue that the observed brittleness is rooted in the absence of formal semantic constraints in current data-driven approaches. They conclude that merely scaling models or training on more code will not resolve these issues. Instead, reliable code reasoning demands formal guarantees that current LLMs cannot provide. The paper positions these findings as a call to action for the research community to develop hybrid approaches that integrate formal methods with machine learning to improve robustness and security against adversarial code mutations.