Sense and Sensitivity: Examining the Influence of Semantic Recall on Long Context Code Reasoning

1. Problem Statement

Large Language Models (LLMs) are increasingly deployed to understand massive codebases, with context windows now supporting millions of tokens. However, a critical gap exists in understanding how these models process long code contexts. It is unclear whether models genuinely understand the operational semantics (what the code does when executed) of code snippets buried in long contexts, or if they rely on pattern matching shortcuts (memorized algorithms or superficial correlations) to solve tasks.

Current benchmarks often conflate two distinct capabilities:

1. Lexical Recall: The ability to locate and reproduce code verbatim.
2. Semantic Recall: The ability to understand the operational semantics of the code to predict execution behavior.

The authors argue that existing evaluations may mask failures in semantic recall because models can solve tasks using memorized patterns rather than reasoning about the specific provided code. Furthermore, the "Lost-in-the-Middle" phenomenon (where performance degrades when information is centrally located) has been observed for retrieval, but its impact on semantic understanding remains unquantified.

2. Methodology

Core Concepts

• Lexical vs. Semantic Recall: The paper distinguishes between retrieving code (Lexical) and understanding its execution logic (Semantic).
• Semantic Recall Sensitivity: A new metric defined as the degree to which a task requires understanding specific code details versus allowing pattern-matching shortcuts. High sensitivity means the task cannot be solved without analyzing the specific provided code.

Experimental Design

The authors evaluated 10 state-of-the-art LLMs (including GPT-4.1, DeepSeek Coder, Llama 3.3, and others) across three tasks:

1. Lexical Recall: Function-level retrieval (reproducing code verbatim) within long contexts containing distractor code.
2. Semantic Recall (Standard): Input/Output prediction on CRUXEval, a popular code reasoning benchmark.
3. Semantic Recall (High Sensitivity): A new task called SemTrace.

Key Experimental Manipulations:

• Positional Variation: Target code was embedded at 11 different positions within contexts containing 20 to 80 distractor functions (ranging from ~4k to ~16k tokens).
• Counterfactual Measurement: To measure "Semantic Recall Sensitivity," the authors systematically removed lines of code from CRUXEval functions.
  • Hypothesis: If a model relies on true semantic recall, accuracy should drop sharply (exponentially) as lines are removed (similar to a Python interpreter). If it relies on pattern matching, accuracy should degrade gradually.
• SemTrace Design: A task where models predict the output of a Python function performing random, unpredictable arithmetic operations on a list. The operations are randomized, preventing the use of memorized algorithmic patterns.

3. Key Contributions

1. Distinction of Recall Types: The paper establishes that frontier LLMs achieve near-perfect, position-independent lexical recall but suffer severe, position-dependent failures in semantic recall.
2. Semantic Recall Sensitivity Metric: Introduced a counterfactual method to quantify how much a benchmark relies on specific code understanding vs. memorized patterns. They demonstrated that popular benchmarks like CRUXEval have low sensitivity, allowing models to "cheat" via pattern matching.
3. SemTrace Benchmark: Proposed a new high-sensitivity task that isolates semantic recall. It revealed that current evaluations drastically underestimate the fragility of code understanding in long contexts.

4. Key Results

Lexical vs. Semantic Dissociation

• Lexical Recall: Frontier models maintained >95% accuracy in retrieving code verbatim regardless of position (start, middle, or end).
• Semantic Recall (CRUXEval): Showed moderate degradation when code was in the middle ("Lost-in-the-Middle"). Median accuracy dropped by 53.36% compared to boundaries.
• Semantic Recall (SemTrace): Showed severe degradation. Median accuracy dropped by 92.73% when code was centrally positioned. Some models (e.g., Codestral 22B, Gemma 3 27B) dropped to 0% accuracy in the middle of long contexts.

Sensitivity Analysis

• CRUXEval Low Sensitivity: When lines were removed from CRUXEval functions, models showed gradual degradation (only ~44-60% accuracy loss at 50% line removal). This indicates models were compensating for missing code using pre-trained knowledge of common algorithms (e.g., sorting, string manipulation) rather than reading the specific code.
• SemTrace High Sensitivity: The task design forced models to rely on the specific provided lines. The sharp drop in accuracy confirmed that the models were failing to semantically recall the specific operations when the context was long and the code was central.

GPT-4.1 Anomaly and Memorization

• GPT-4.1 appeared to have perfect semantic recall on SemTrace (2-digit arithmetic) regardless of position.
• Discovery: Further testing with 4–6 digit arithmetic revealed that GPT-4.1's performance became position-dependent. This proved GPT-4.1 had memorized 2-digit arithmetic from pretraining, effectively turning the semantic task into a lexical retrieval task. When the arithmetic exceeded its memorization capacity, it exhibited the same fragility as other models.

Generalization

• The dissociation between lexical and semantic recall was consistent across Python, JavaScript, and PHP.
• Smaller models exhibited the same qualitative patterns but with more severe degradation and higher variance.

5. Significance and Implications

• Evaluation Flaws: Current benchmarks (like CRUXEval) significantly underestimate the failure rates of LLMs in long-context code understanding because they permit pattern-matching shortcuts.
• Production Risks: In real-world scenarios (e.g., debugging novel project-specific code or detecting subtle vulnerabilities), pattern matching is insufficient. The severe "Lost-in-the-Middle" effect on semantic recall suggests models may fail to understand critical logic buried in large codebases, leading to hallucinations or missed vulnerabilities.
• Future Directions: The paper calls for new benchmarks with high semantic recall sensitivity (like SemTrace) to accurately assess a model's true ability to reason about operational semantics in long contexts. It also suggests that the "Lost-in-the-Middle" effect is not just a retrieval issue but a fundamental limitation in how models integrate and process procedural information.

In summary, the paper reveals a critical gap: while LLMs can "find" code anywhere in a long document, they often fail to "understand" what that code does when it is buried in the middle, relying instead on memorized patterns that fail when the specific context is novel or complex.