Technical Summary: MAVEN - Evidence-State Rewards for Long-Context Reasoning

1. Problem Statement

Advanced Large Language Models (LLMs) increasingly support long context windows, yet the ability to accept long inputs does not guarantee effective reasoning over them. Models often suffer from "lost in the middle" phenomena, premature reliance on partial evidence, and an inability to synthesize information distributed across distant parts of the context.

Existing Reinforcement Learning (RL) approaches for long-context reasoning typically rely on outcome-only rewards (verifying the final answer) or static evidence-identification rewards (rewarding the selection of gold chunks). The authors argue these methods are insufficient because they treat evidence as isolated chunks or a final extracted set. They fail to capture the dynamic, state-dependent nature of long-context reasoning, where:

• A segment may only become valuable after another is found.
• A plausible document may need to be discarded once later evidence reveals it as a distractor.
• Individually insufficient pieces may jointly bridge a reasoning gap.

Current rewards encourage premature commitment to locally plausible evidence paths rather than the iterative process of building, revising, and synthesizing an evolving evidence state.

2. Methodology: MAVEN

The authors propose MAVEN (Marginal-Value Evidence Navigation), a reinforcement learning framework that treats long-context reasoning as stateful evidence-memory navigation. Instead of rewarding isolated extraction, MAVEN defines an answer-conditioned evidence-state value and assigns rewards to specific action-level state transitions.

2.1 Framework and Actions

The policy model $\pi_\theta$ interacts with an editable evidence memory $E_k$ before generating a final answer. The model generates a trajectory of high-level actions:

1. Add (<add>): Adds a quoted or referenced evidence span to the memory.
2. Drop (<drop>): Removes an irrelevant, redundant, or misleading evidence piece.
3. Link (<link>): Explains how two evidence pieces jointly support the answer (synthesis).
4. Answer (<answer>): Produces the final response.

2.2 Value Function

MAVEN utilizes a frozen verifier model $p_\psi$ to compute an evidence-state value $V_\psi(Q, E)$. This value measures the relative reduction in the negative log-likelihood (NLL) of the gold answer $a^*$ given the question $Q$ and the current evidence memory $E$, compared to having no evidence:
$$V_\psi(Q, E) = \frac{\ell_\psi(Q, \emptyset) - \ell_\psi(Q, E)}{\ell_\psi(Q, \emptyset) + \epsilon}$$
A positive value indicates the evidence makes the correct answer more predictable.

2.3 Action-Level Rewards

MAVEN assigns process rewards to specific action spans within the GRPO (Group Relative Policy Optimization) framework:

• Add Reward ($\rho_{add}$): Combines online Conditional Answer Information Gain (CAIG) and hindsight credit.
  • CAIG: Rewards immediate improvement in the evidence state upon adding a new item.
  • Hindsight Credit: If an item is part of the final kept evidence set $E_F$, it receives credit based on how much the final value would drop if that item were removed (a Shapley-style approximation). This ensures early, necessary evidence is not under-credited in multi-hop reasoning.
• Drop Reward ($\rho_{drop}$): Rewards the removal of evidence if it improves the answer-conditioned evidence state (i.e., removing a distractor). It penalizes dropping useful evidence.
• Link Reward ($\rho_{link}$): Rewards the explicit synthesis of complementary evidence. It uses a pairwise synergy score to measure if two pieces are more valuable together than separately.
• Answer Reward ($\rho_{ans}$): A binary reward based on substring matching with the ground truth to ensure task success.

2.4 Training Procedure

Training occurs in two stages:

1. Cold-Start SFT: Supervised fine-tuning on valid action trajectories to teach the model the editable memory interface and action grammar.
2. RL Training: The model is optimized using GRPO with the action-local rewards defined above. The objective preserves process-level credit assignment, allowing a trajectory with a bad "add" followed by a corrective "drop" to receive appropriate negative and positive advantages respectively.

3. Key Contributions

1. Identification of Limitations: The authors identify that existing long-context RL rewards often supervise static evidence extraction rather than evidence-state transitions, leading to premature anchoring on incorrect paths.
2. MAVEN Framework: Proposes an editable evidence-memory framework that defines an answer-conditioned evidence-state value. It trains LLMs to add, link, drop, and answer through stateful, action-local process rewards.
3. Empirical Validation: Demonstrates consistent gains across different model families (Llama, Qwen) and benchmarks, providing diagnostic evidence that MAVEN improves evidence sufficiency (coverage of gold evidence) and reduces distractor retention (keeping misleading evidence).

4. Experimental Results

The authors evaluated MAVEN on Llama-3.1-8B, Qwen2.5-14B, and Qwen3-30B-A3B across LongBench v2, LongReason, and RULER.

• Performance: MAVEN consistently outperformed all baselines, including outcome-only RL and outcome-plus-evidence-identification baselines. On LongBench v2, MAVEN improved the strongest baseline by +3.5 to +4.0 points across model sizes.
• Long-Context Specifics: Gains were particularly pronounced in long-context splits (e.g., 128K context), suggesting the method is crucial when answers depend on distributed information.
• Diagnostic Metrics:
  • Evidence Sufficiency: MAVEN achieved >85% sufficiency, significantly higher than baselines.
  • Distractor Retention: MAVEN significantly reduced the percentage of kept evidence that corresponded to distractors, demonstrating the model's ability to revise its memory.
• Ablation Studies:
  • Removing the drop reward led to the highest distractor retention.
  • Removing the link reward lowered evidence sufficiency.
  • Removing the add reward caused the largest drop in overall score.
  • A variant using rewards computed only from the final evidence set (ignoring intermediate steps) performed substantially worse, confirming the necessity of action-local process rewards.
• Generalization: Evaluation on MMLU-Pro showed that MAVEN training did not degrade general short-context reasoning capabilities.

5. Significance and Claims

The paper claims that effective long-context RL requires optimizing evidence-state transitions rather than merely optimizing evidence extraction. By treating reasoning as a navigation process through an editable memory, MAVEN enables models to:

• Recover from adding misleading evidence (via drop rewards).
• Synthesize complementary information (via link rewards).
• Recognize the necessity of early evidence even when its immediate value is low (via hindsight credit).

The authors position MAVEN as complementary to retrieval, agentic workflows, and architectural context extensions, specifically targeting the bottleneck of learning to build and revise the evidence state used for reasoning.

Limitations Acknowledged:

• MAVEN relies on gold answers during training to compute verifier-based values, limiting direct applicability to tasks with verifiable answers.
• The interface assumes evidence can be represented as bounded text spans.
• Diagnostic annotations were constructed for controlled analysis, and broader human evaluation of evidence quality remains future work.