Technical Summary: In-the-Flow Agentic System Optimization for Effective Planning and Tool Use

Problem Statement

While outcome-driven reinforcement learning (RL) has significantly advanced reasoning in Large Language Models (LLMs), prevailing tool-augmented approaches face critical limitations. Most existing methods train a single, monolithic policy that interleaves reasoning and tool calls under a full context. This architecture scales poorly with long reasoning horizons and diverse toolsets, often leading to training instability and weak generalization to unseen scenarios.

Conversely, agentic systems offer a promising alternative by decomposing tasks across specialized modules (e.g., planners, executors, critics). However, most current agentic systems are training-free, relying on handcrafted logic or static prompting heuristics. Those that do employ training typically use offline supervised fine-tuning (SFT) or preference optimization on static datasets. These off-policy approaches are decoupled from the live dynamics of multi-turn interactions, failing to capture how modules should adapt to evolving tool outputs or recover from early mistakes. Consequently, agentic systems struggle with sparse rewards, brittle adaptation, and inefficient orchestration in dynamic environments.

Methodology: AGENTFLOW and Flow-GRPO

The authors introduce AGENTFLOW, a trainable, "in-the-flow" agentic framework designed to optimize planning and tool use directly within the multi-turn interaction loop.

1. The AGENTFLOW Architecture

AGENTFLOW coordinates four specialized modules through a shared, evolving memory ($M$) and a toolset ($K$):

• Planner ($P$): A trainable policy ($\pi_\theta$) that formulates sub-goals, selects tools, and retrieves context from memory.
• Executor ($E$): Invokes the selected tool and returns execution results ($e_t$).
• Verifier ($V$): Evaluates the validity of the execution and whether the accumulated memory is sufficient to solve the query, producing a binary signal ($v_t$).
• Generator ($G$): Produces the final solution upon termination.

Unlike latent thought processes in monolithic models, AGENTFLOW maintains an explicit, deterministic, and structured memory that records the reasoning process. This ensures transparency, controllable behavior, and bounded context growth. The system operates as a multi-turn Markov Decision Process (MDP) where the planner is optimized on-policy inside the live loop.

2. Flow-based Group Refined Policy Optimization (Flow-GRPO)

To address the long-horizon credit assignment problem inherent in sparse, trajectory-level rewards, the authors propose Flow-GRPO. This on-policy algorithm transforms the complex multi-turn RL challenge into a sequence of tractable single-turn updates:

• Trajectory-Level Reward Broadcasting: Instead of assigning brittle intermediate rewards, the system assigns a single, verifiable final-outcome reward ($R(\tau)$) to the entire trajectory. This reward is broadcast to every turn ($t$) within the rollout.
• Single-Turn Decomposition: By broadcasting the global success signal to every turn, the multi-turn optimization is effectively decomposed into independent single-turn policy updates. At each turn, the planner has access to the full historical context (via memory) and receives a consistent reward signal aligned with global success.
• Group-Normalized Advantages: To stabilize training and reduce variance, the algorithm uses group-normalized advantages. For a group of $G$ parallel rollouts, the advantage for a specific trajectory is calculated by normalizing its reward against the mean and standard deviation of the group's rewards.

The objective function maximizes the expected return over parallel rollouts, utilizing a clipped surrogate objective (similar to PPO) with a KL-divergence penalty against a reference policy.

Key Contributions

1. AGENTFLOW Framework: A trainable agentic system that directly optimizes its planner inside the multi-turn loop. By coordinating specialized modules via an evolving memory, it enables adaptive long-horizon planning and robust tool orchestration, bridging the gap between monolithic reasoning models and static agentic systems.
2. Flow-GRPO Algorithm: An on-policy, outcome-driven algorithm that converts multi-turn RL into a sequence of single-turn updates. It solves the credit assignment problem by broadcasting a verifiable trajectory-level outcome to every turn and stabilizing learning with group-normalized advantages.
3. Comprehensive Empirical Validation: Extensive experiments across ten diverse benchmarks demonstrate that AGENTFLOW with a 7B-parameter backbone outperforms specialized baselines and even larger proprietary models.

Experimental Results

The authors evaluated AGENTFLOW on ten benchmarks spanning knowledge-intensive search, agentic reasoning, mathematical reasoning, and scientific reasoning.

• Performance Gains: AGENTFLOW (7B backbone) achieved average accuracy gains of 14.9% on search tasks, 14.0% on agentic tasks, 14.5% on mathematical tasks, and 4.1% on scientific tasks compared to top-performing baselines.
• Comparison to Proprietary Models: Notably, the 7B-scale AGENTFLOW system surpassed the ~200B-parameter GPT-4o across all domains, with gains ranging from 8.2% to 18.0%.
• Ablation Studies:
  • Training Strategy: Replacing the frozen planner with a stronger model (GPT-4o) yielded only modest gains. Conversely, offline SFT on GPT-4o trajectories caused a catastrophic performance collapse (19.0% drop), whereas in-the-flow Flow-GRPO training yielded a 17.2% gain over the frozen baseline.
  • Scaling: Performance improvements were consistent when scaling the backbone from 3B to 7B parameters. Additionally, increasing the maximum allowed turns ($T_{max}$) from 3 to 10 consistently improved outcomes, validating the system's ability to adapt turn allocation to problem complexity.
• Qualitative Analysis: Case studies revealed that Flow-GRPO enables the planner to:
  • Optimize tool selection (e.g., shifting from general search to specific web search for domain-specific tasks).
  • Recover from errors (e.g., correcting variable naming issues in code execution loops).
  • Discover new solution pathways (e.g., switching tools strategically after initial failures).

Significance and Claims

The paper claims that in-the-flow optimization is crucial for effective agentic systems. By training the planner directly within the live multi-turn loop using Flow-GRPO, the system learns to handle long-horizon workflows, adapt to dynamic tool feedback, and recover from compounding errors—capabilities that offline training methods fail to capture.

The authors assert that their approach demonstrates that a carefully designed agentic system, even with a relatively small 7B-parameter backbone, can deliver superior and more efficient performance compared to substantially larger monolithic LLMs. This suggests that the architecture and training methodology (specifically the combination of modular decomposition and on-policy flow optimization) are more critical factors for complex reasoning success than sheer model scale alone. The work establishes a new state-of-the-art for agentic systems, proving that trainable, in-the-flow optimization leads to improved planning, enhanced tool-calling reliability, and positive scaling trends.