Technical Summary: Positional Failures in Long-Context LLMs

Problem Statement

While retrieval-oriented long-context benchmarks (e.g., Needle-in-a-Haystack, RULER) have established position-controlled evaluation as a standard, mainstream reasoning benchmarks (e.g., LongBench, SCROLLS, L-Eval) largely fail to control or report the positional placement of target tasks within long contexts. This creates a structural evaluation gap: current benchmarks report aggregated performance on natural documents where task-relevant information inherits its position from the source, making it impossible to disentangle model failure from task position.

Furthermore, an audit of four recent flagship long-context model releases (DeepSeek-V4-Pro, MiMo-V2.5-Pro, Kimi-K2.6, and GLM-5.1) reveals that vendor result tables prioritize agentic and coding benchmarks (e.g., SWE-Bench, Terminal-Bench) while omitting position-controlled reasoning metrics like NIAH or RULER from their main performance tables. Consequently, positional vulnerabilities that may grow with context length remain uncharacterized in both academic benchmarks and industry evaluations.

Methodology: Context Rot Evaluation (CRE)

To address this gap, the authors propose Context Rot Evaluation (CRE), a controlled framework that treats task position, filler content, and context length as three independent variables.

• Experimental Design: A full factorial design varying:
  • Position: End (target immediately before response), Middle (target surrounded by filler), Middle-Twice (target repeated in middle), and Middle-Dup (target in middle + copy at end).
  • Filler Content: Three types ranging from high to low overlap with the target task:
    • with_solutions (ws): GSM8K problems with worked solutions (high overlap).
    • questions_only_v2 (qo_v2): GSM8K problems without solutions (intermediate overlap).
    • neutral_text: Unrelated Wikipedia/news prose (low overlap).
  • Context Length: Three tiers: 8K, 32K, and 64K tokens.
• Tasks: GSM8K (math word problems) and ARC-Challenge (science multiple choice).
• Models: Nine long-context LLMs evaluated across two rounds:
  • Initial Set (5 models): Qwen 2.5-7B-Instruct, MiMo-v2-Flash, GLM-4.7-FlashX, DeepSeek-V3.2 (reasoning mode), Kimi k2.5.
  • Round 2 (4 newer releases): DeepSeek-V4-Pro, MiMo-V2.5-Pro, Kimi-K2.6, GLM-5.1.
• Diagnostic Probes: The framework includes "middle_dup" (duplicating the task at the end) to test if models retain task-solving ability under favorable positioning, and "filler-answer matching" to quantify interference.

Key Results

1. Positional Vulnerability Escalates with Context Length

Models exhibit significant accuracy drops when the target task moves from the end to the middle of the context, with severity increasing as context length grows.

• MiMo-v2-Flash: Under with_solutions filler at 64K, accuracy drops from 96% (end) to 8% (middle), a −88 percentage point (pp) drop.
• Qwen 2.5-7B: Drops from 94% (end) to 0% (middle) at 64K under with_solutions filler (−94pp).
• DeepSeek-V3.2: Shows near-immunity under with_solutions filler (0pp drop) but becomes vulnerable under questions_only_v2 filler (−22pp at 64K).
• Kimi k2.5: Shows minimal drops under with_solutions but significant drops under questions_only_v2.

2. Filler Content Interacts with Position

The nature of the filler content significantly modulates positional failure.

• Structured Filler (with_solutions): Causes the most severe drops for vulnerable models (e.g., MiMo-v2-Flash, Qwen).
• Unanswered Questions (questions_only_v2): Produces drops across all models, including those immune to with_solutions (e.g., DeepSeek-V3.2 drops −22pp at 64K).
• Neutral Text: Generally benign for most models at 8K, but Qwen exhibits significant drops (−30pp at 64K) even with neutral filler, suggesting a model-specific vulnerability to context length regardless of content.

3. Error Analysis: Filler-Answer Interference

A diagnostic analysis reveals that 76% of middle-position errors match the gold answer of a surrounding filler question, compared to only 22% at the end position. This suggests that filler-answer interference is a dominant error mode when the task is buried in the middle of the context.

4. Diagnostic Probes and Model Recovery

• Middle-Dup Probe: When a copy of the target task is placed at the end (in addition to the middle), accuracy for all nine models at 8K recovers to within ±4pp of the end-position baseline. This confirms that models retain the reasoning capability but fail specifically due to the middle position.
• Round 2 Vendor Upgrades: Newer releases (e.g., MiMo-V2.5-Pro) show reduced drops under with_solutions filler (e.g., −32pp vs. −88pp for the previous version) but still exhibit significant drops (−16pp to −56pp) under questions_only_v2 filler.

5. Cross-Domain Validation

On ARC-Challenge, all nine models show directional end-to-middle drops (range −6pp to −40pp), though statistical significance is limited by sample size (N=50). Only Qwen's drop is Bonferroni significant, suggesting the phenomenon generalizes but with varying magnitude across domains.

Significance and Claims

The paper claims to expose a structural evaluation gap in current reasoning benchmark design and vendor evaluation practices. Its primary contributions are:

1. Systematic Audit: Demonstrating that position-aware evaluation is mature for retrieval but largely absent from mainstream reasoning benchmarks and vendor disclosure tables.
2. Evidence of Joint Factors: Showing that task position, filler content, and context length jointly shape reasoning failures. Specifically, that "middle-position drops" are not merely a function of context length but are heavily modulated by filler type and model architecture.
3. Model-Dependent Spectrum: Establishing that positional vulnerability is not uniform; some models (e.g., DeepSeek-V3.2 under specific conditions) are nearly immune, while others (e.g., Qwen, MiMo-v2-Flash) suffer catastrophic failures in the middle of long contexts.
4. Evaluation Recommendations: Proposing that future evaluations must report per-position accuracy, test interior positions (not just boundaries), disclose full inference stacks, and include diagnostic probes to distinguish positional effects from general capability loss.

The authors conclude that without controlling for task position, benchmarks cannot measure positional vulnerabilities that grow with context length, leaving a critical blind spot in the assessment of long-context LLMs.