Robustness and Reasoning Fidelity of Large Language Models in Long-Context Code Question Answering

1. Problem Statement

While Large Language Models (LLMs) have shown promise in software engineering tasks, their ability to reason over long code contexts (thousands of lines across multiple files) remains unproven under realistic conditions. Previous benchmarks, such as LongCodeBench, primarily focused on:

• Single-language scope: Mostly Python, neglecting legacy (COBOL) and other modern (Java) ecosystems.
• Fixed task formulations: Relying heavily on multiple-choice questions (MCQs), which may allow models to exploit superficial cues (lexical overlap, positional patterns) rather than demonstrating genuine code comprehension.
• Lack of robustness testing: Failing to systematically evaluate how models handle irrelevant information (distractors) or variations in input formatting.

The core problem is that current evaluations do not distinguish between recognition (selecting the right answer from a list) and generation (synthesizing an answer from scratch), nor do they assess the models' resilience to adversarial inputs or context length scaling.

2. Methodology

The authors conducted a systematic study extending LongCodeBench with controlled ablations and new datasets.

A. Dataset Expansion (LongContextCodeQA)

The study introduced a multilingual benchmark suite:

• Python: Retained from the original LongCodeBench.
• COBOL: Two new datasets derived from enterprise systems:
  • OPPSCAL: Public Medicare Pricer code (complex pricing algorithms).
  • Internal IBM: High-volume transaction processing logic (CICS, database interactions).
• Java: Curated from four top GitHub repositories (Elasticsearch, Cassandra, Dubbo, Kafka), covering API behavior, exception handling, and design patterns.
• Scale: Context lengths ranging from 32k to 1,024k (1M) tokens.

B. Controlled Perturbations (Robustness Evaluation)

To probe reasoning fidelity beyond raw accuracy, the authors introduced three experimental settings:

1. Shuffled Options: Randomizing the order of MCQ answers to test for positional or lexical biases.
2. Without Options (Open-Ended): Removing answer choices entirely, forcing the model to generate the answer directly. This measures the Recognition-Generation Gap.
3. Needle-in-a-Haystack (NiH):
   • Relevant Needle: Inserting the correct code snippet at the Start, Middle, or End of the context to test positional recall and attention mechanisms.
   • Irrelevant Needle: Inserting adversarial, unrelated code fragments at varying depths to test distractor robustness.

C. Models Evaluated

A diverse set of state-of-the-art models was tested, including GPT-4o, Gemini-2.5 (Flash/Pro), Claude-4.5-Sonnet, Llama-3.1/3.3, Mistral, Qwen, and Granite.

3. Key Contributions

1. LongContextCodeQA Benchmark: A multilingual extension of LongCodeBench featuring COBOL and Java datasets derived from real-world enterprise and open-source systems.
2. Robustness Suite: A novel evaluation framework incorporating option shuffling, open-ended generation, and distractor injection to expose model fragility.
3. Empirical Analysis: A comprehensive study revealing consistent failure modes across languages, specifically the gap between recognition and generation, positional biases, and vulnerability to irrelevant information.

4. Key Results

A. The Recognition-Generation Gap

• Observation: All models exhibit a massive performance drop when moving from MCQ ("With Options") to Open-Ended ("Without Options") tasks.
• Data:
  • Python: Accuracy drops by 15–35 percentage points. (e.g., Claude-4.5-Sonnet drops from 80.5% to 42.5%).
  • COBOL: The gap is even more severe, with drops of 30–60 percentage points. (e.g., Gemini-2.5-Pro drops from 98.4% to 32.0% at 512k context).
  • Java: The gap is smaller but still significant, suggesting better generative capabilities for modern languages compared to legacy ones.
• Implication: Models rely heavily on pattern matching against provided options rather than deep reasoning or reconstructive generation.

B. Context Length and Scaling

• Non-Uniform Scaling: Increasing context window size does not guarantee improved reasoning. Many models show performance degradation or inconsistency as context grows (e.g., Llama-3.3-70B dropping from 85.1% at 64k to 65.8% at 128k).
• Language Dependence: COBOL shows extreme fragility. Models struggle significantly more with long contexts in legacy languages compared to Python or Java, likely due to lower pretraining data volume and different syntactic structures.

C. Positional Bias ("Lost in the Middle" vs. "Recency Bias")

• Recency Bias: Contrary to the classic "Lost in the Middle" phenomenon, the study observed a strong recency bias, particularly in COBOL. Models performed significantly better when the relevant information was at the End of the context compared to the Start or Middle.
• Start Degradation: Performance for information at the Start of the context (especially in COBOL) crashed significantly (e.g., Granite-4-8B dropped to 15.56% accuracy for Start positions in COBOL without options).

D. Distractor Robustness

• Fragility: Models are highly sensitive to irrelevant information. When adversarial "needles" were inserted, performance degraded, especially in open-ended settings.
• Constrained Search Helps: The "With Options" setting consistently outperformed "Without Options" by 20%+, indicating that providing a constrained search space significantly aids retrieval but masks the model's inability to filter noise independently.

5. Significance and Conclusion

This paper fundamentally challenges the assumption that "longer context windows" equate to "better code reasoning."

• Fragility of Current Models: Even frontier models (e.g., Gemini-2.5-Pro, GPT-4o) are brittle when required to generate answers without scaffolding or when faced with irrelevant code.
• Legacy Code Risk: The inability of models to robustly handle long-context COBOL poses a critical risk for enterprise software maintenance and modernization efforts.
• Evaluation Reform: The authors argue that standard MCQ benchmarks are insufficient. Future evaluations must prioritize open-ended generation, positional robustness, and distractor resistance to truly assess an LLM's reasoning fidelity in software engineering workflows.

Final Takeaway: Long-context code QA is far from solved. Current models excel at recognition tasks but fail at generative reasoning, particularly in legacy languages and complex, noisy environments.