Technical Summary: Agent-as-a-Router

1. Problem Statement

Real-world users often have access to multiple Large Language Models (LLMs) from different providers, each excelling in distinct domains (e.g., algorithm design, data science, or code completion) but none dominating all tasks. Current coding agents typically rely on a single, fixed LLM for all tasks, a design that prioritizes provider-side predictability over user-centric cost-efficiency and task-level quality.

The core problem is automatic model routing: determining which model should handle each incoming coding task to maximize performance while minimizing cost. Existing routing methods treat this as a static, one-off classification problem, often employing language models as routing policies. However, preliminary experiments reveal a significant performance gap between these static routers and a "per-task oracle" (which knows the best model for every specific task). The paper identifies that the bottleneck for existing routers is not a lack of reasoning capability, but rather an information deficit: static routers fail to accumulate execution-grounded experience during deployment.

2. Methodology

2.1. Diagnosis: Information Deficit

The authors conducted an ablation study using a zero-shot LLM router (powered by Claude Sonnet 4.6).

• Vanilla Router: Scored 41.41% AvgPerf.
• +Perf Stats: When provided with prior performance statistics from a held-out probing set, the score rose to 47.74% (+15.3% relative gain), surpassing even the best heuristic router (47.50%) that encoded the same static priors.
• Conclusion: The primary limitation is the lack of dynamic, execution-grounded information, not the reasoning capacity of the router model itself.

2.2. The Agent-as-a-Router Framework

To address the information deficit, the authors propose Agent-as-a-Router, a framework that formalizes routing as a Context-Action-Feedback (C-A-F) loop. Unlike static routers, this framework evolves over a stream of tasks:

1. Context ($c_i$): The router observes the task prompt, metadata, and accumulated history (Memory) from previous loops.
2. Action ($a_i$): The router selects a model from the candidate pool.
3. Feedback ($f_i$): The selected model's output is executed in a sandbox, and a verifier generates a score and cost.
4. Update: The feedback is memorized and merged into the context for the next task ($c_{i+1}$).

This formulation relates to a contextual multi-armed bandit problem, where the natural metric for evaluation is cumulative regret (the accumulated gap between the router's performance and the per-task oracle).

2.3. ACRouter Instantiation

The framework is instantiated as ACRouter, comprising three core modules:

• Orchestrator: Makes routing decisions by integrating the DimensionBest prior, top-10 historical neighbors (retrieved via cosine kNN from Memory), and task metadata. It uses a lightweight, fine-tuned Qwen3.5-0.8B model combined with heuristic rules via weighted voting.
• Verifier: Evaluates model outputs in a sandboxed environment. It aggregates signals (AST parsing, execution results, LLM-as-Judge) into a unified performance score ($u_i \in [0, 1]$).
• Memory: An online vector store keyed by task embeddings. It logs the chosen model, performance, cost, and verification traces. It uses a FIFO-bounded store (20K entries) to enable fine-grained, context-aware retrieval of past successes and failures.

2.4. CodeRouterBench

To evaluate routing on streaming tasks, the authors introduce CodeRouterBench, a benchmark comprising:

• ~10,000 tasks across 10 coding dimensions (9 in-distribution single-turn tasks and 1 out-of-distribution agentic-programming dimension).
• 8 Frontier LLMs (including Claude Opus 4.6, GPT-5.4, Qwen3-Max, Kimi-K2.5, etc.).
• Execution-Verified Scores: Tasks are scored via sandboxed execution or LLM-as-Judge, providing a ground-truth observation matrix for regret-based comparison.

3. Key Contributions

1. Framework: The proposal of Agent-as-a-Router, which formalizes model routing as a C-A-F loop, enabling routers to actively close the information gap through execution-grounded feedback.
2. Artifacts:
   • ACRouter: A concrete instantiation of the framework with Orchestrator, Verifier, and Memory modules.
   • CodeRouterBench: A standardized evaluation environment (~10K tasks, 8 LLMs) designed specifically for regret-based router comparison on streaming tasks, including an OOD agentic-programming testbed.
3. Findings:
   • Information Deficit: The routing bottleneck is information access, not reasoning. Providing performance statistics yields a 15.3% relative gain.
   • Performance: ACRouter achieves the lowest cumulative regret on in-distribution tasks and generalizes effectively to out-of-distribution (OOD) agentic-programming tasks.
   • Static Limitations: Lightweight static routers (trained classifiers, heuristics) perform reasonably on in-distribution tasks but fail significantly on OOD tasks, whereas ACRouter maintains strong performance.

4. Experimental Results

Experiments were conducted on both in-distribution (ID) and out-of-distribution (OOD) task streams.

• In-Distribution Performance:

  • ACRouter achieved the highest average performance (49.98%) and the lowest cumulative regret (205.5), outperforming the static "DimensionBest" heuristic (47.50%) and all trained policy baselines.
  • It also demonstrated superior cost-efficiency (Perf/$ ratio of 3.79) compared to always using the strongest single model (Always-Opus 4.6, which had a ratio of 1.29).

• Out-of-Distribution (OOD) Generalization:

  • On the agentic-programming testbed (176 tasks), ACRouter achieved 62.50% AvgPerf, significantly outperforming the Always-Opus baseline (57.14%) and all other routers.
  • Static classifiers (e.g., LogReg, RouteLLM-BERT) collapsed on OOD tasks, with performance dropping to 8.93%–21.43%, even lower than random selection.
  • ACRouter's ability to accumulate verified experience during the stream allowed it to adapt to the new task distribution, whereas static methods relying on frozen priors failed to generalize.

• Cost Analysis:

  • While ACRouter incurs overhead for the Memory and Verifier modules, it remains more cost-effective than always selecting the most expensive model (Opus 4.6) while delivering higher quality.
  • The "Always-Kimi-K2.5" strategy offered the highest cost-efficiency (Perf/$ 12.62) but at a substantial cost to absolute performance (36.66% AvgPerf).

5. Significance and Claims

The paper claims that actively closing the information gap through execution-grounded feedback is a general principle for building agentic systems that must route among heterogeneous tools or models.

• Shift in Paradigm: The work moves routing from a static classification problem to a dynamic, self-adaptive process (C-A-F loop).
• Practical Utility: It demonstrates that users can achieve near-oracle performance and cost-efficiency by dynamically routing tasks, rather than relying on a single "best" model.
• Generalizability: The C-A-F loop is presented not just as a routing mechanism but as a broader decision-making paradigm applicable to tool selection, API endpoint selection, and effort allocation.
• Modesty: The authors acknowledge limitations, such as the inability to observe provider-side cache hit rates (making cost estimates relative rather than absolute) and the use of a reduced step limit for OOD tasks to maintain budget tractability. They note that while their current instantiation uses an LLM policy with kNN memory, alternative parameter-level memory techniques remain to be explored.

In summary, the paper argues that for agentic coding systems, the future lies not in finding a single dominant model, but in building adaptive routers that learn from execution feedback to select the right tool for the right job.