Technical Summary: ReContext – Recursive Evidence Replay as LLM Harness for Long-Context Reasoning

1. Problem Statement

Despite the increasing context window sizes of Large Language Models (LLMs), a critical bottleneck remains in long-context utilization rather than mere access. Recent studies indicate that while models can accept inputs of 128K tokens or more, they frequently fail to consistently locate and utilize relevant evidence present within the input, leading to errors and hallucinations. This "context harnessing" gap suggests that the bottleneck is not just in reading the context, but in dynamically managing, organizing, and updating the most relevant information during the reasoning process.

Existing approaches attempt to address this through:

• Attention Intervention: Modifying low-level model behavior (e.g., attention logits), which is invasive and requires changing backbone logic.
• Retrieval and External Memory: Adding external retrieval systems, which may lose fine-grained details.
• Compression: Shortening the input via pruning or summarization, which risks discarding necessary information.

These methods often rely on external components, alter the model's decoding logic, or sacrifice the integrity of the original context. There is a need for a mechanism that preserves full access to the original input while dynamically surfacing relevant evidence to guide generation without training or external memory.

2. Methodology: RECONTEXT

The authors propose RECONTEXT (Recursive Evidence Replay as LLM Harness for Long-Context Reasoning), a training-free inference method designed to improve long-context reasoning. The core philosophy is to separate evidence organization from answer generation.

Core Workflow

1. Evidence Sifting (Internal Signal Extraction):

   • Given a long context $C$ and a question $q$, the model performs an initial read pass.
   • It utilizes model-internal relevance signals (specifically, attention weights from the question-conditioned suffix tokens to context tokens) to score the relevance of context tokens.
   • A scoring function aggregates attention from a small set of cue tokens (the question suffix) across selected layers and heads to compute relevance scores $r_i$ for each token.

2. Evidence Materialization:

   • Selected high-scoring tokens are mapped back to their containing sentences or local spans in the original text.
   • These spans are extracted to form a grounded evidence pool $E$. Crucially, this pool is constructed from the original text, not generated by the model, ensuring factual grounding.

3. Recursive Replay:

   • The method employs a recursive selection process over $R$ rounds.
   • In each round $j$, the prompt consists of the full original context $C$, the accumulated evidence pool $E^{(j-1)}$ from previous rounds, and the question $q$.
   • The model reads this augmented prompt. The presence of the evidence pool conditions the hidden states, allowing the model to compute new relevance scores based on the updated state.
   • New relevant spans are selected from the original context (not the replayed text) and appended to the evidence pool if they are not already present.
   • This process repeats for a fixed number of rounds, building an ordered scaffold of evidence.

4. Final Generation:

   • The final answer is generated from a prompt containing the full original context, the final organized evidence pool, and the question.
   • The full context remains accessible throughout; the evidence pool serves as an emphasis mechanism, not an exclusion filter.

Theoretical Framework

The paper frames RECONTEXT through the lens of associative memory:

• Context: A memory store.
• Question: A retrieval cue.
• Attention: A proxy for cue-trace association.
• Replay: Reactivation of selected traces near generation time.

The authors provide a theoretical proof (Theorem 1) demonstrating monotonic improvement. They show that under specific assumptions (orthogonal embeddings, relevant query), recursively adding the most relevant evidence to the prompt strictly increases the cosine similarity between the model's hidden representation and the answer embedding at each step.

3. Key Contributions

1. RECONTEXT Method: A training-free inference method that converts prompt-internal relevance signals into an explicit, recursive evidence pool while preserving the full original context. It avoids modifying backbone parameters or pruning inputs.
2. Theoretical Analysis: An associative-memory interpretation of the method, formalizing evidence selection as cue-trace association and replay as trace reactivation, accompanied by a proof of monotonic improvement in hidden representation alignment.
3. Comprehensive Evaluation: Extensive experiments on eight long-context datasets (128K context length) across three backbone models (Qwen3-4B, Qwen3-8B, Llama3-8B).

4. Experimental Results

• Performance: RECONTEXT consistently outperforms strong baselines (Vanilla, AttnSharp, DySCO, A-MEM, DAC) across all three backbones.
  • It achieves the best average rank on all three models.
  • It improves mean accuracy over the Vanilla baseline from 0.24 to 0.30, a 24.6% relative gain.
  • On Qwen3-4B, it achieves the best score on every reported metric (e.g., improving NQ Accuracy from 0.02 to 0.08).
• Robustness:
  • Shorter Contexts: Under a 64K context budget, RECONTEXT remains in the top two on every metric, showing gains are not tied to a specific context length.
  • Thinking Mode: When "thinking" (chain-of-thought) is enabled, RECONTEXT remains the strongest or near-strongest method, indicating its utility persists even when the model performs explicit intermediate reasoning.
• Efficiency:
  • RECONTEXT is slower than Vanilla due to the read-and-replay stages but is substantially faster than methods that alter decoding logic (e.g., DySCO).
  • Memory overhead is minimal (adding fewer than 128 evidence tokens), keeping GPU memory consumption similar to Vanilla.
• Ablation Studies:
  • Recursion Depth ($R$): Moving from 1 to 2 rounds significantly improves performance. Optimal depth varies by task (e.g., $R=4$ for NQ, $R=2$ for PopQA).
  • Token Budget ($K$): Larger candidate sets ($K=32$) benefit PopQA and InfMC, while smaller sets ($K=8$) are optimal for NQ, suggesting a trade-off between recall and noise.
  • Source Selection: Selecting evidence tokens strictly from the original context outperforms selecting from the full replay prompt, validating the design choice to keep the scaffold as a conditioning mechanism rather than a new source of tokens.

5. Significance and Claims

The paper claims that RECONTEXT demonstrates that long-context inference can be improved not only by extending context windows or compressing inputs, but by better organizing the evidence already available in the prompt.

• Separation of Concerns: By separating evidence organization from answer generation, RECONTEXT allows the model to ground its answers in a compact set of supporting sentences while retaining the safety net of the full context.
• Training-Free Efficiency: It offers a practical, training-free alternative to complex retrieval systems or invasive attention modifications, leveraging the model's own internal signals.
• Generalizability: The method shows consistent gains across different model families and scales, suggesting that the "context harnessing" problem is a general limitation of current LLM architectures that can be mitigated via inference-time scaffolding.

The authors acknowledge limitations, noting that the method requires access to internal attention signals (limiting use with closed-source APIs) and introduces inference latency compared to direct decoding, though it remains training-free and memory-efficient.