Technical Summary: SkillFlow: Flow-Driven Recursive Skill Evolution for Agentic Orchestration

1. Problem Statement

Recent advancements in Large Language Model (LLM)-based agentic systems have enabled the automation of complex tasks through task orchestration. However, existing orchestration methods face three critical challenges:

1. Strategy Collapse: Reinforcement learning (RL) methods based on REINFORCE-family objectives (e.g., GRPO) tend to converge to a single reward-maximizing mode. This eliminates diverse, equally effective strategies, making agents brittle when executor capabilities evolve or when multiple valid paths exist.
2. High Gradient Variance and Opaque Credit Assignment: Terminal-only rewards create long credit-assignment horizons. When a multi-step trajectory succeeds or fails, it is unclear which specific step was responsible, leading to high variance in policy gradients and opaque attribution.
3. Unguided Skill Evolution: Frameworks with dynamic skill libraries often rely on heuristic triggers, fixed schedules, or direct "LLM-as-judge" prompting to update their libraries. These approaches lack principled training signals to determine when to evolve, what skills to create or prune, and where critical decision gaps lie.

2. Methodology: SkillFlow

SkillFlow is a flow-based framework designed to automate task orchestration through multi-turn interaction while recursively evolving its own skill library. It operates via a Supervisor-Executor paradigm where a trainable Supervisor ($\pi_\theta$) interacts with a structured environment containing a dynamic skill library and a frozen Executor.

2.1 Core Architecture

• Orchestration State Graph: The task process is modeled as a Directed Acyclic Graph (DAG) where nodes represent interaction histories ($H_t$) and edges represent orchestration actions.
• Dynamic Skill Library: The environment maintains a library of "atomic tips" (short, self-contained strategic guidance). This library co-evolves with the policy but remains fixed within each training phase to ensure a constant action space.
• Hindsight Backward Policy: A learned backward policy ($P_\phi$) conditions on the "hindsight state" ($H_{t-1} \oplus o^{exec}_t$), incorporating execution feedback unavailable to the forward policy at the time of decision.

2.2 Tempered Trajectory Balance (TTB)

The core of SkillFlow is Tempered Trajectory Balance (TTB), a regression-based flow-matching loss. Unlike REINFORCE, which pushes probability mass onto a single best trajectory, TTB drives the sampling probability of a trajectory to be proportional to its reward:
$$\pi^*(\tau | q) \propto \tilde{R}(\tau)^\beta$$
where $\tilde{R}(\tau)$ is the smoothed reward and $\beta$ controls the diversity-quality tradeoff.

The TTB loss is defined as the squared, length-normalized residual of the Trajectory Balance condition:
$$L_{TTB}(\tau) = \left( \frac{\Delta(\tau)}{T} \right)^2$$
$$\Delta(\tau) = \log Z_\theta(q) + \sum_{t=1}^T \log \pi_\theta(a_t | r_t, H_{t-1}) - \beta \log \tilde{R}(\tau) - \sum_{t=1}^T \log P_\phi(a_t | H_{t-1} \oplus o^{exec}_t)$$
At convergence ($\Delta(\tau) = 0$), the forward policy samples trajectories in proportion to their rewards, preserving strategic diversity.

2.3 Zero-Cost Credit Assignment

The same flow objective jointly trains the backward policy, enabling per-step credit assignment at zero additional inference cost. By comparing the forward policy's decision with the hindsight backward policy's evaluation, SkillFlow computes Step Importance ($I(t)$):
$$I(t) = \frac{\pi_\theta(a_t | r_t, H_{t-1})}{P_\phi(a_t | H_{t-1} \oplus o^{exec}_t)}$$
Large deviations in $I(t)$ indicate decisions where the assessment shifted significantly after execution, effectively flagging the critical steps that drove success or failure.

2.4 Flow-Driven Recursive Skill Evolution

SkillFlow closes the loop from training signals to autonomous capability growth using flow diagnostics to drive a curation operator $\Phi$:

• When to Evolve: Evolution is triggered when the TTB residual ($\Delta(\tau)^2$) saturates against a plateau, indicating the current library limits the policy's expressiveness.
• What to Create/Prune: Skills are ranked by their Skill Marginal Flow ($\hat{F}(s)$), which measures the average flow a skill attracts. Skills with persistently negative contributions are pruned; those with high flow are retained.
• Where to Evolve: New skills are generated at "trigger steps" identified by high Step Importance ($I(t)$) where successful and failed trajectories from the same query diverge. A Skill Creator ($\Psi$) synthesizes new atomic tips from these decision gaps.

3. Key Contributions

1. TTB for Orchestration: Introduction of a flow-matching loss that preserves diverse orchestration strategies proportional to reward, avoiding the mode collapse inherent in REINFORCE-family objectives.
2. Zero-Cost Credit Assignment: A jointly learned backward policy that provides transparent, per-step attribution of trajectory outcomes without requiring additional rollouts or inference.
3. Principled Skill Evolution: A recursive mechanism that autonomously expands, prunes, and refines the skill library based on flow diagnostics (TTB residuals, step importance, and marginal flow), eliminating the need for heuristic triggers or LLM-as-judge prompting.
4. Empirical Validation: Comprehensive evaluation demonstrating that SkillFlow outperforms direct LLMs, RL baselines (GRPO, Tree-GRPO), and existing skill-evolution methods across 14 benchmarks.

4. Experimental Results

SkillFlow was evaluated on 14 datasets spanning four categories: Question Answering (HotpotQA, TriviaQA, MuSiQue, NQ-Open), Mathematical Reasoning (AIME 2026, MATH-Hard, GPQA Diamond), Interactive Decision Making (WebShop, ALFWorld, ScienceWorld, Mind2Web), and Code Generation (SWE-bench, HumanEval).

• Performance: SkillFlow significantly outperformed all baselines. On In-Distribution (IID) benchmarks, it achieved an average Exact Match (EM) of 94.14% (vs. 73.34% for Qwen3.5-9B and 90.63% for AgentFlow). On Out-of-Distribution (OOD) benchmarks, it maintained a strong lead with an average EM of 83.99% (vs. 76.96% for AgentFlow).
• Diversity and Robustness: The framework demonstrated superior strategy diversity, particularly in tasks like WebShop and ALFWorld where REINFORCE baselines suffered from strategy collapse.
• Transferability: SkillFlow improved performance across various LLM backbones (including Qwen3.5, Claude, and Gemini), with weaker backbones showing the most significant gains, suggesting the method compensates for base reasoning limitations.
• Efficiency: SkillFlow achieved these results with lower computational costs (fewer tokens and time) compared to other skill-evolution baselines, attributed to the efficiency of flow-signal-driven evolution.

5. Significance and Claims

The paper claims that SkillFlow establishes a foundation for self-evolving, flow-based agentic orchestration. Its significance lies in unifying orchestration training and recursive skill evolution under a single theoretical framework (GFlowNets).

• Diversity Preservation: By sampling trajectories proportional to reward rather than collapsing to a single mode, SkillFlow maintains a repertoire of strategies, making agents more robust to changes in tool capabilities or task variations.
• Transparent Learning: The zero-cost credit assignment mechanism provides interpretability, allowing the system to identify exactly which decisions were pivotal, a feature often missing in terminal-reward RL.
• Autonomous Growth: The recursive skill evolution mechanism closes the loop between training signals and capability growth, allowing the agent to autonomously identify and fill decision gaps without human intervention or heuristic scheduling.

The authors note that while SkillFlow effectively amplifies orchestration and tool use, it relies on the underlying backbone's long-context memory and reasoning capacity; it does not replace the need for core reasoning skills but rather optimizes how those skills are composed and evolved.