Technical Summary: MAVEN – Improving Generalization in Agentic Tool Calling

1. Problem Statement

The paper addresses the challenge of generalization in agentic tool-calling systems. While Large Language Models (LLMs) demonstrate strong performance on individual benchmarks, they often lack robustness in long-horizon settings where agents must decompose complex requests, preserve intermediate states, coordinate multiple tools across domains, and verify partial results.

Current evaluation methods often rely on final-answer correctness, which can mask failures in reasoning processes. Models may adapt to specific dataset formats or interaction patterns (overfitting) rather than developing generalizable compositional reasoning skills. The authors argue that existing benchmarks (e.g., BFCL v3, TauBench) do not sufficiently stress-test process fidelity, intermediate state management, or explicit verification, leading to a gap between partial reasoning quality and end-to-end task success.

2. Methodology

MAVEN: Modular Agentic Verification and Execution Network

The authors propose MAVEN, a lightweight, symbolic reasoning scaffold designed to augment LLMs without requiring additional training. MAVEN operates as a structured layer that enforces a three-stage pipeline:

1. Context Buffering: The system extracts and organizes relevant information from the conversational input into a compact, short-lived representation. This preserves salient facts and intermediate reasoning necessary for downstream processing.
2. Action Synthesis: Using the buffered representation, the system generates atomic, testable task descriptions aligned with user objectives. This stage includes a bounded refinement process to ensure clarity and supports early termination if no further action is needed.
3. Invocation Generation: The system produces machine-interpretable actions compatible with the execution environment. Crucially, this stage explicitly separates reasoning from execution to minimize unsafe side effects and maintain auditability.

The workflow (Algorithm 1) involves a loop where the agent synthesizes subtasks, selects tools, executes them, and stores outputs with diagnostics and provenance in a buffer. If verification fails, the agent revises the subtask or invocation using the stored context before proceeding.

MAVEN-Bench: A Stress-Test Benchmark

To evaluate these capabilities, the authors introduce MAVEN-Bench, a benchmark focused on multi-step scientific reasoning (mathematics and physics) with external tools. Key features include:

• Parametric Instantiation: The core corpus consists of 100 canonical problem templates across calculus, algebra, mechanics, and electromagnetism. These are parameterized to create diverse numerical regimes, algebraic forms, and verification requirements, including adversarial perturbations (e.g., ill-conditioning, near-degenerate points).
• Model Context Protocol (MCP): A defining component where intermediate artifacts (symbolic expressions, numerics, diagnostics) are treated as first-class objects. They are stored under explicit step identifiers, allowing agents to persist, query, and reason about intermediate results.
• Evaluation Metrics: The benchmark evaluates agents on multiple axes:
  • Sub-question accuracy: Correctness at individual verification checkpoints.
  • Tool selection accuracy: Appropriateness of the chosen primitive.
  • Trace fidelity: Alignment with canonical reference traces.
  • Verification score: Execution and correct interpretation of explicit checks (e.g., unit consistency, solver residuals).
  • Final-answer correctness: Symbolic equivalence and numeric tolerance.

3. Key Contributions

The paper makes three primary contributions:

1. MAVEN-Bench: A process-aware benchmark for tool-augmented mathematical and physical reasoning that emphasizes intermediate state preservation and explicit verification over simple final-answer correctness.
2. Evaluation Protocol: A rigorous protocol that records tool traces, intermediate artifacts, and verification behaviors, utilizing an automated LLM judge with a fixed rubric to assess tool use, correctness, and approach.
3. MAVEN Framework: An evaluation of the MAVEN scaffold, demonstrating that lightweight, verification-centered scaffolds can significantly improve compositional reasoning and long-horizon robustness in open-weight models.

4. Results

Performance on General Benchmarks

MAVEN was evaluated on established benchmarks (BFCL v3, TauBench, Tau2Bench, AceBench). When instantiated on top of the GPT-OSS-120b base model, MAVEN consistently outperformed the base model, particularly in multi-turn and agentic settings. The improvements suggest that structured decomposition and intermediate verification reduce error propagation in complex tasks.

Performance on MAVEN-Bench

In direct runs on MAVEN-Bench:

• Accuracy Improvement: MAVEN improved the accuracy of the GPT-OSS-120b base model from 48% to 71% without additional training.
• Competitiveness: MAVEN (running on GPT-OSS-120b) achieved a score of 71.0%, outperforming several frontier proprietary models (e.g., GPT-5 at 32.0%, o4-mini at 38.0%) and remaining competitive with others (e.g., Claude-Sonnet-4.5 at 70.0%).
• Cost Efficiency: The open-weight backbone used by MAVEN has an estimated cost ratio of roughly 1/10 compared to proprietary baselines.
• Robustness: Analysis of accuracy as a function of reasoning steps (Figure 4) showed that while accuracy generally degrades as problem complexity increases, models augmented with MAVEN exhibited a slower degradation rate, indicating better long-horizon robustness.

Observed Failure Modes

The study identified recurring failure modes in current systems, including:

• Incorrect tool selection (e.g., choosing numerically unstable solvers).
• Failure to perform explicit intermediate verification (omitting sign validation or derivative checks).
• Protocol violations (deviating from single-call constraints).
• Numerical instability in adversarial parameter regimes.

5. Significance and Claims

The paper argues that process-aware evaluation is essential for assessing the reliability of agentic systems. The results reveal a significant gap between performance on standard benchmarks and robust reasoning in settings requiring sustained decomposition and state management.

The authors claim that:

• Lightweight scaffolds like MAVEN can strengthen compositional reasoning and reduce error accumulation in long-horizon tool-use settings.
• Explicit verification and structured decomposition are critical for handling complex, multi-step tasks where small errors in tool selection or numerical stability compound over time.
• MAVEN-Bench effectively exposes failures (such as incorrect tool choice or missing verification) that are often invisible when evaluating only final answers.

The work concludes that reliable tool-using agents should not only produce correct final answers but also follow auditable, verifiable, and compositional reasoning processes. The authors position MAVEN as a reasoning augmentation rather than a new foundation model, suggesting that such scaffolds can motivate more rigorous evaluation of agents "in the wild."

Note: The paper acknowledges limitations, including the focus on mathematical/physical domains, the potential disadvantage of constrained execution protocols for models optimized for flexible reasoning, and the reliance on LLM-based judges despite rubric calibration.