💬 NLP

FlowBot: Inducing LLM Workflows with Bilevel Optimization and Textual Gradients

This paper introduces FlowBot, a data-driven framework that automatically induces and optimizes LLM workflows by formulating the process as a bilevel optimization problem solved through modular textual gradients, achieving performance competitive with human-crafted baselines.

Original authors: Hongyeon Yu, Young-Bum Kim, Yoon Kim

Published 2026-04-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Hongyeon Yu, Young-Bum Kim, Yoon Kim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a team of specialized robots to solve a very difficult puzzle. In the past, humans had to sit down and manually write a strict instruction manual for every single robot, deciding exactly what order they should work in and what specific words to say to each one. This is slow, expensive, and hard to scale.

The paper introduces FLOWBOT, a new method that lets the robots teach themselves how to organize and talk to each other, without needing a human to write the manual first.

Here is how it works, using a simple analogy:

The Two-Level Coaching System

Think of FLOWBOT as a coaching system with two levels: a Head Coach and a Position Coach.

1. The Head Coach (The Outer Loop): "Fixing the Game Plan"
The Head Coach looks at the big picture. They don't worry about the specific words a player says; they worry about the structure of the game.

The Problem: Maybe the team is trying to solve a math problem by first trying to draw a picture, which is the wrong order. Or maybe they are missing a step entirely, like "checking the score."
The Fix: The Head Coach looks at where the team failed and says, "Okay, we need to add a step to check the score before we try to draw the picture," or "Let's remove the drawing step entirely."
In the Paper: This is the Outer Loop. It optimizes the "workflow sketch." It decides how many steps there are, what order they happen in, and whether to add or remove tools (like search engines).

2. The Position Coach (The Inner Loop): "Refining the Playbook"
Once the Head Coach sets the game plan, the Position Coach works with individual players (the specific LLM calls) to perfect their specific instructions.

The Problem: The player knows what to do (e.g., "search for facts"), but they are doing it poorly. Maybe they are hallucinating (making things up) or missing a key detail.
The Fix: Instead of just saying "do better," the Position Coach uses a special trick called Textual Gradients.
- Imagine the final answer is wrong. The Position Coach asks an AI judge: "Why did this happen?"
- The judge gives a text comment: "You made a mistake because you didn't check your source."
- This comment is passed backwards to the previous player, who says, "Oh, I need to make sure my source is clear for the next person."
- This continues all the way back to the first player.
In the Paper: This is the Inner Loop. It uses "Textual Backpropagation." Just like a neural network uses math to adjust numbers, FLOWBOT uses natural language feedback to adjust the text prompts of each step.

The "Textual Gradient" Magic

In traditional computer science, if a calculation is wrong, the computer uses math to figure out exactly how much to tweak the numbers to fix it. This is called "backpropagation."

LLMs (Large Language Models) don't understand math in the same way; they understand language. So, FLOWBOT invented Textual Gradients.

Instead of a number like +0.5, the system gets a sentence like: "Your answer was wrong because you assumed X, but the evidence shows Y. Please check your sources next time."
The system passes this sentence backwards through the chain of robots. Each robot reads the feedback, understands what went wrong in the steps after them, and rewrites their own instructions to prevent that error in the future.

What Did They Find?

The researchers tested FLOWBOT on many different tasks, like answering complex trivia questions, writing code, and following strict instructions.

It works from scratch: Unlike other methods that need a human to give them a good starting plan, FLOWBOT can start with a blank slate and figure out the best workflow on its own.
It beats the competition: It performed as well as (or better than) systems where humans spent a lot of time hand-crafting the perfect workflow.
It saves money: Because it learns so efficiently, it requires fewer "API calls" (which cost money) to find the right solution compared to other automated methods.

The Bottom Line

FLOWBOT is like a self-improving assembly line. It doesn't just tweak the instructions on the machines; it also rearranges the machines themselves. By using natural language feedback to "backpropagate" errors, it automatically discovers the best way to organize a team of AI models to solve complex problems, removing the need for humans to be the architects of every single workflow.

1. Problem Statement

Large Language Models (LLMs) are increasingly used in "compound AI systems"—structured workflows involving multiple LLM calls, tools, and intermediate reasoning steps to solve complex tasks. However, current approaches to building these workflows face two major bottlenecks:

Human Reliance: Most workflows are hand-crafted by humans, which is not scalable across diverse tasks and domains.
Limitations of Existing Automation: Recent automated approaches (e.g., AFlow, ADAS) often optimize prompts within a fixed workflow structure or search for workflow structures using code-based representations without fine-grained, layer-by-layer optimization. They lack a mechanism to simultaneously optimize both the high-level workflow architecture and the specific instructions (prompts) for each component.

The core research question is: How can we automatically induce and optimize both the structure and the individual components of an LLM workflow in a data-driven way?

2. Methodology: FLOWBOT

The authors propose FLOWBOT, a framework that formulates workflow induction as a bilevel optimization problem solved using textual gradients.

A. Problem Formulation

An LLM workflow $W$ is defined as a chain of $K$ LLM calls: $W = [a_1, \dots, a_K]$ .

Outer Loop (Structure): Optimizes the high-level sketch $W$ (the sequence, type, and composition of calls).
Inner Loop (Parameters): Optimizes the detailed prompt $\theta_i$ for each specific call $a_i$ .
Objective: Minimize the expected loss $L(f_W(x), y)$ over a dataset $D$ .

B. Textual Gradients

Since LLM parameters are discrete text strings, traditional backpropagation is impossible. FLOWBOT uses textual gradients, where an LLM generates natural language feedback describing how to change an input or prompt to reduce loss.

C. The Optimization Process

The framework operates in two nested loops:

1. Inner Loop: Layer-wise Textual Backpropagation
Given a fixed workflow structure $W$ , the system optimizes the prompts $\{\theta_i\}$ :

Forward Pass: The workflow processes input $x$ to produce output $x_K$ .
Loss Gradient ( $g_K$ ): An LLM analyzes the final output, ground truth, and loss to generate a textual gradient $g_K$ (feedback on what went wrong).
Chain Rule Propagation: Gradients are propagated backward layer-by-layer. For layer $k$ , an LLM takes the current input $x_k$ , the next layer's output $x_{k+1}$ , and the downstream gradient $g_{k+1}$ to compute $g_k$ . This answers: "How should step $k$ 's output change to help step $k+1$ perform better?"
Prompt Update: An optimizer LLM aggregates gradients from a batch of samples and updates the prompt $\theta_i$ to address the identified errors while preserving successful patterns.

2. Outer Loop: Workflow Sketch Optimization
Once the inner loop converges (or periodically), the system optimizes the structure $W$ itself:

The current workflow sketch, execution traces, and aggregated loss are fed to an LLM.
The LLM generates a workflow gradient ( $g_W$ ) suggesting structural changes: adding steps (e.g., "add entity disambiguation"), removing redundant steps, merging steps, or reordering.
The workflow sketch is updated based on these suggestions.

D. Initialization

The process starts with a "zero-shot" workflow (a single LLM call). The outer loop is triggered immediately to generate an initial multi-step sketch, followed by the generation of initial prompts for each step.

3. Key Contributions

Bilevel Optimization for Workflows: The first framework to jointly optimize the topology (structure) and parameters (prompts) of LLM workflows using a bilevel approach, drawing inspiration from Neural Architecture Search (NAS).
Textual Backpropagation: Extends the concept of textual gradients (previously used for single prompts) to deep, multi-layer workflows, enabling modular, layer-by-layer error attribution and correction.
Data-Driven Induction: Demonstrates that complex, multi-step workflows (including tool usage) can be discovered from scratch without human priors on the structure.

4. Experimental Results

The authors evaluated FLOWBOT on 10 diverse benchmarks (QA, reasoning, code generation, privacy, etc.) against strong baselines.

vs. Prompt Optimization (Fixed Structure): FLOWBOT outperformed GEPA (a state-of-the-art prompt optimizer) on retrieval-heavy tasks like HotpotQA (72.80 vs 69.00) and HoVer (63.20 vs 51.67), proving that optimizing structure is crucial for complex tasks.
vs. Automated Workflow Generation: FLOWBOT outperformed AFlow (which uses Monte Carlo Tree Search on code graphs) on 4 out of 6 benchmarks (e.g., HumanEval: 93.74 vs 94.70; GSM8K: 96.01 vs 86.16).
Efficiency: FLOWBOT required significantly fewer API calls and lower costs than AFlow to reach peak performance (e.g., 54.7K calls vs 338K calls for HotpotQA).
Qualitative Analysis: The induced workflows were found to be sensible and often mirrored human-designed patterns (e.g., separating retrieval, entity disambiguation, and verification).
Ablation Studies:
- Removing the bilevel structure (treating it as a single layer) caused significant performance drops.
- Removing layer-wise backpropagation (updating all prompts at once) also degraded performance, highlighting the importance of modular, chain-rule-based feedback.

5. Significance and Impact

Paradigm Shift: Moves LLM system design from "hand-crafted pipelines" to "automatically induced systems," reducing the engineering burden for deploying complex AI agents.
Generalizability: The approach is model-agnostic and works with various open-weight and proprietary LLMs.
Interpretability: The textual gradients provide a human-readable audit trail of why a workflow failed and how it was improved, offering insights into the reasoning process of the system.
Future Direction: The paper demonstrates a proof-of-concept for inducing a unified workflow that can handle multiple task types (routing math vs. code vs. reading comprehension) within a single dynamic structure, paving the way for general-purpose autonomous agents.

In summary, FLOWBOT bridges the gap between neural architecture search and LLM prompting, providing a robust, data-driven method to automatically construct and refine sophisticated AI workflows.