💬 NLP

FlyRoute: Self-Evolving Agent Profiling via Data Flywheel for Adaptive Task Routing

FlyRoute is a self-evolving profiling framework that leverages a data flywheel to dynamically update agent capability descriptions from real-world traffic, significantly improving adaptive task routing accuracy over static baselines through targeted exploration and evidence distillation.

Original authors: Rongjun Li, Ziyu Zhou, Yihang Wu

Published 2026-05-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Rongjun Li, Ziyu Zhou, Yihang Wu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a busy, high-tech office building where a receptionist (the Router) is responsible for directing every visitor's question to the right expert in the building.

In a traditional office, the receptionist relies on a static directory written by the building managers. This directory says, "Dr. Smith handles cloud issues," and "Engineer Lee handles mobile phones." The problem is, the experts are constantly learning new skills, buying new tools, and changing how they work. But the directory never updates. So, the receptionist keeps sending questions to the wrong people because their old job descriptions are outdated.

FlyRoute is a new, self-improving system that fixes this by turning the receptionist into a smart, learning machine. Here is how it works, using simple analogies:

1. The "Self-Evolving" Directory

Instead of relying on a static list, FlyRoute builds a living profile for every expert based on what they actually do.

The Old Way: You write a job description once, and it stays there forever, even if the expert learns to speak a new language or use a new tool.
The FlyRoute Way: Every time an expert successfully solves a problem, that success is saved as a "proof of skill." Over time, the system reads through thousands of these successful cases and rewrites the expert's job description to match their current reality, not their old resume.

2. The "Data Flywheel" (The Engine)

Think of this system as a bicycle with a flywheel. Once you get it moving, it gains momentum and keeps spinning on its own.

Step 1: Dispatch. A user asks a question. The system guesses which expert is best.
Step 2: The Quality Gate. If the expert gives a good answer, the system saves that "Question + Good Answer" pair in a special "Success Library." If the answer is bad, it's discarded.
Step 3: Distillation. Periodically, the system reads through the Success Library and writes a fresh, concise summary of what that expert is actually good at.
Step 4: Better Routing. The next time a question comes in, the router uses this fresh summary (plus examples from the library) to make a smarter guess.
The Result: Better routing leads to more good answers, which leads to more data, which leads to even better summaries. The system gets smarter the more it is used.

3. The "Smart Explorer" (Uncertainty-Driven Exploration)

A common mistake in these systems is to either:

Only send questions to the "famous" experts (who are already known to be good), leaving new experts with no chance to prove themselves.
Send questions to everyone randomly, which is a waste of time and money.

FlyRoute uses a Smart Explorer strategy. Imagine a scout looking for new talent.

If an expert has very few "Success Library" entries, the system knows their profile is "uncertain."
The system then asks: "Is this new question something this uncertain expert might be able to handle?" (It checks if the words in the question match what the expert has done before).
If the answer is "maybe," the system sends the question to that expert just to test them.
If the expert succeeds, the system adds that proof to their library. If they fail, it learns they aren't the right fit for that type of question.
This ensures the system doesn't waste time on experts who clearly can't help, but it does give new or updated experts a fair chance to show off their new skills.

4. The Results

The researchers tested this on a real-world enterprise system with four types of experts (Cloud, AI, Hardware, and Mobile OS).

Starting Point: With just a tiny bit of initial data (5 examples per expert), FlyRoute was already 78% accurate, beating the standard "zero-shot" system (which had no examples) at 72%.
After Learning: As the system processed over 7,000 real user questions and let the "flywheel" spin, the accuracy jumped to 89.8%.

The Bottom Line

FlyRoute solves the problem of "stale job descriptions." It admits that experts change, and instead of waiting for a human to update the directory, it watches the experts work, learns from their successes, and constantly rewrites their profiles to ensure the next question goes to the right person. It turns a static list into a self-improving, living map of who knows what.

Technical Summary: FlyRoute

Problem Statement

In Multi-Agent Systems (MAS) deployed for enterprise tasks, a critical bottleneck exists in agent routing: the mechanism that assigns incoming queries to the most suitable expert agent. Current routing paradigms rely on static agent profiles registered at deployment time. These profiles, typically consisting of developer-written capability descriptions and fixed exemplars, fail to adapt to the dynamic reality of production environments where agents evolve through prompt updates, tool integrations, and model switches.

The authors identify two primary failure modes of static profiling:

Registration Inaccuracy: Developers often struggle to provide precise natural language descriptions of an agent's actual boundaries, and representative exemplars are difficult to curate.
Post-Deployment Drift: As agents are updated to handle new scenarios or integrate new tools, their registered profiles become stale, causing the router to make decisions based on outdated information.

Existing solutions, such as performance-based routing (balancing cost/latency) or task-based routing using static classifiers, do not address the need for continuously evolving capability profiles derived from live traffic.

Methodology: FlyRoute

FlyRoute is a self-evolving agent profiling framework designed to transform deployment interactions into a data flywheel that continuously updates routing evidence. The system operates through a closed loop comprising three core components:

1. Adaptive Agent Profiling

Instead of relying solely on static descriptions, FlyRoute maintains a dynamic profile $P_i$ for each agent $a_i$ , consisting of:

Seed Description ( $d_{seed}$ ): The initial, potentially incomplete, developer-provided text.
Learned Description ( $d_{learn}$ ): A refined capability summary distilled from empirical evidence.
Success Example Store ( $E_i$ ): A collection of high-quality query-response pairs $(q, r, s)$ where the response $r$ passed a quality gate.

The Process:

Cold Start: New agents register with minimal data (name, optional seed description, and $k$ seed examples). The router initially relies on the seed description and sparse retrieval.
Quality Gating: When an agent generates a response, an LLM-as-Judge evaluates the quality. Only pairs scoring above a threshold $\theta$ are added to the success store.
Capability Distillation: Periodically (every $M$ new successes), an LLM is prompted to synthesize the accumulated success examples and the seed description into a concise, updated Learned Description. This description captures empirical competence boundaries that may differ from the original developer intent.

2. Uncertainty-Driven Exploration

To efficiently build profiles without broadcasting every query to all agents (which is cost-prohibitive), FlyRoute employs a targeted exploration policy. It balances exploitation (routing to agents with strong profiles) and exploration (probing agents with uncertain profiles).

The exploration score $V^+_{explore}$ for a query $q$ and agent $a_i$ is calculated as:
$V^+_{explore}(q, a_i) = U(a_i) \cdot R(q, a_i) \cdot (1 + \beta \cdot N(q, a_i))$

Where:

Profile Uncertainty ( $U$ ): Inversely proportional to the size of the success store $|E_i|$ . Agents with few examples receive higher exploration pressure.
Relevance ( $R$ ): A lexical relevance score (via BM25) between the query and the agent's existing success store. This ensures exploration only occurs for queries plausibly within the agent's domain.
Novelty ( $N$ ): A redundancy penalty ( $1 - R$ ) that prioritizes queries that are relevant but not near-duplicates of existing evidence, ensuring efficient data collection.

3. Profile-Aware Routing

The routing decision is made by an LLM router that receives:

Global Context: The agent's active description (either the distilled $d_{learn}$ or the seed $d_{seed}$ ).
Local Context: Top- $k$ BM25-retrieved success examples from the agent's store that are lexically similar to the current query.

This dual-context approach allows the router to leverage both the broad capability boundaries learned from the stream and specific lexical patterns from recent successes.

Key Contributions

Problem Formalization: The paper identifies and formalizes the challenge of self-evolving agent profiling, shifting the paradigm from static registration to continuous updating via deployment interactions.
Uncertainty-Driven Exploration Policy: The authors propose a novel strategy that combines profile uncertainty, lexical relevance, and redundancy-aware novelty to efficiently target underspecified agents, avoiding the waste of random broadcasting.
Empirical Validation: The framework is validated on a proprietary enterprise dataset, demonstrating significant improvements in routing accuracy over static baselines.

Experimental Results

The authors evaluated FlyRoute on a dataset of 7,211 real routed queries across four enterprise domains: Cloud Services, AI Accelerator, Server Hardware, and Mobile OS.

Cold Start Performance: Even with only 5 seed queries per agent and no streaming data, FlyRoute (using seed descriptions + BM25 retrieval) achieved 78.04% accuracy, outperforming a zero-shot LLM router (72.57%) by 5.47 percentage points.
Flywheel Effect: After streaming all 7,211 training queries through the system, accuracy rose to 89.83%.
- This represents a +17.26 pp gain over the zero-shot baseline.
- This represents a +11.79 pp gain over the cold-start state.
Exploration Strategy: Ablation studies showed that the uncertainty-driven exploration policy significantly outperformed pure exploitation, $\epsilon$ -greedy, and random broadcasting strategies, particularly in the early stages of profile building.
Ablation: Removing the quality gate (LLM-as-Judge) or the exploration component resulted in measurable accuracy drops, confirming the necessity of both high-quality evidence filtering and targeted data collection.

Significance and Claims

The paper claims that FlyRoute demonstrates the viability of data-driven, self-evolving agent management. Its primary significance lies in showing that:

Profile Retrieval is Powerful: Even with minimal seed data, retrieving examples from a success store strengthens routing decisions immediately.
Continuous Learning is Feasible: Systems can autonomously discover and refine agent capabilities from live traffic, correcting for the "stale profile" problem inherent in static systems.
Efficient Exploration: It is possible to build robust profiles without exhaustive broadcasting by using uncertainty and relevance signals to guide where new evidence is collected.

The authors note limitations, including the reliance on proprietary data (limiting public reproducibility), the dependence on the quality of the LLM-as-Judge for feedback, and the lack of explicit drift detection mechanisms for sudden agent changes. However, the results suggest that a closed-loop data flywheel is a critical component for scaling adaptive multi-agent systems in enterprise environments.