A Cost-Aware, Paired Protocol for Auditing Dynamic Tool Synthesis in Agentic Video Question Answering

Problem Statement

Agentic Video Question Answering (VideoQA) systems currently rely on fixed tool libraries. While these agents dynamically select and sequence tools during inference, recurring procedures must be rebuilt from primitive operations for every new question. This redundancy creates significant inference overhead.

Synthesizing composite tools (reusable sequences of primitives) could amortize this planning and selection cost. However, assessing the value of such synthesis is difficult using standard metrics. Final-answer accuracy alone ignores inference effort; a system might achieve higher accuracy by expending significantly more computational resources, or conversely, improve efficiency at the cost of accuracy. Existing evaluations fail to reveal how expanding the action space shifts the trade-off between correctness and cost.

Methodology

1. Dynamic-SAGE Framework

The authors propose Dynamic-SAGE, a framework that extends the existing SAGE agentic VideoQA system. Unlike SAGE, which uses a static set of tools, Dynamic-SAGE dynamically expands its reusable action space before evaluation.

• Offline Synthesis Pipeline: Using a held-out set of 150 VideoQA questions, a multi-agent system synthesizes composite tools:
  • Signature Agent: Identifies recurring multi-step reasoning patterns and proposes tool interfaces (name, description, inputs/outputs).
  • Implementation Agent: Composes existing primitive operations (e.g., ground event, transcribe speech, analyze) into executable Python functions. It is constrained to use only existing primitives, adding no new external models or capabilities.
  • Verification Agent: Validates candidates for safety, contract compliance, and correctness, rejecting duplicates, malformed code, or tools that attempt to commit a final answer (leaving answer commitment to the orchestrator).
• Inference: During evaluation, the agent (orchestrated by GPT-4o) selects from the union of the original static tools ($T_s$) and the synthesized dynamic tools ($T_d$). A synthesized tool appears as a single action to the orchestrator but internally executes a fixed sequence of primitives.

2. Cost-Aware Paired Evaluation Protocol

To rigorously assess the impact of tool synthesis, the authors introduce a cost-aware, paired protocol.

• Paired Comparison: Instead of comparing aggregate accuracy, the protocol pairs the Static-SAGE and Dynamic-SAGE systems on the same input questions.
• Six-Group Classification: Each paired outcome is categorized based on joint correctness (Static vs. Dynamic) and the change in visible tool calls ($\Delta e_i$):
  1. $G_{safe}$: Both correct, Dynamic uses fewer calls.
  2. $G_{neutral}$: Both correct, no change in calls.
  3. $G_{overhead}$: Both correct, Dynamic uses more calls.
  4. $G_{ideal}$: Static wrong, Dynamic correct, fewer calls.
  5. $G_{costly-gain}$: Static wrong, Dynamic correct, same/more calls.
  6. $G_{loss}$: Static correct, Dynamic wrong (treated as a failure regardless of efficiency).
• Significance Testing: The protocol employs McNemar's test for statistical significance and paired bootstrap confidence intervals to report reliability across subgroups.

Key Contributions

1. A Cost-Aware Paired Protocol: A reusable evaluation practice that audits tool-augmented agents by jointly reporting accuracy and cost changes, separating efficiency gains from harmful regressions.
2. Dynamic-SAGE Testbed: A framework that synthesizes, validates, and persistently registers composite tools for reuse on unseen questions without introducing new models or primitive capabilities.
3. Multi-Axis Audit: An empirical analysis on SAGE-Bench revealing that tool synthesis improves accuracy but shifts cost structures rather than reducing them uniformly.

Results

Evaluated on 1,304 samples from SAGE-Bench (excluding the 150-question synthesis holdout):

• Accuracy: Dynamic-SAGE improves overall accuracy by 7.52 percentage points (from 60.43% to 67.94%, $p < 0.001$).
  • Gains are largest on visual questions (+9.29%) and open-ended questions (+9.17%).
  • Gains are statistically neutral for verbal and mixed-modality questions.
  • Improvements scale with difficulty, showing the largest gains on "hard" and "very hard" questions.
• Efficiency & Cost Trade-offs:
  • Reduced Orchestration: Dynamic-SAGE reduces reasoning turns and visible tool calls by approximately 28% (from 3.34 to 2.40 turns).
  • Increased Token/Cost: Because composite tools encapsulate multiple primitives, the total number of primitive operations increases by 31%. Consequently, token usage rises by 34%, and monetary cost increases by 26%.
• Failure Analysis:
  • The majority of accuracy losses occur on hard, open-ended questions where the pipeline performs the most work.
  • Most failures (53%) happen when Dynamic-SAGE uses more tool calls than Static-SAGE, indicating that extended trajectories still reach incorrect answers, rather than premature over-compression.
  • Adoption is skewed: two depth-2 tools account for most invocations, while deeper tools are rarely used and less accurate.

Significance and Claims

The paper claims that the primary contribution is not just the system, but the protocol that reveals a "multi-axis profile" of performance that scalar accuracy comparisons miss.

The authors conclude that expanding the action space via tool synthesis does help, but the benefit is conditional. It shifts the cost burden from orchestration steps (reasoning turns, visible calls) toward per-call token consumption. The system achieves higher accuracy and fewer interaction steps but at a higher monetary and token cost.

The authors modestly note that the current approach is not a universal solution; reliability risks remain, particularly for difficult, open-ended questions. They suggest that future work should focus on improving the quality, distinctiveness, and routing of synthesized tools rather than merely expanding the action space. The paper does not claim to solve VideoQA entirely but provides a rigorous method for auditing the trade-offs of dynamic tool synthesis.