💬 NLP

GFlowRL: Scaling Distribution-Matching RL to Large Language Models

GFlowRL introduces a scalable, stable GFlowNet-style reinforcement learning algorithm for large language models that eliminates the need for a learned partition function by using in-batch Monte Carlo estimates and specialized stabilizers, achieving state-of-the-art performance on math, code, and adversarial benchmarks across both dense and sparse architectures up to 235B parameters.

Original authors: Xiaodong Liu, Michael Xu, Jack W. Stokes, Paul Smolensky, Doug Burger, Jianfeng Gao

Published 2026-07-16

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiaodong Liu, Michael Xu, Jack W. Stokes, Paul Smolensky, Doug Burger, Jianfeng Gao

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a brilliant, super-fast student how to solve the world's hardest puzzles, from complex math problems to writing computer code that never crashes. In the world of artificial intelligence, this student is a "Large Language Model" (LLM), and the method we use to teach them is called Reinforcement Learning. Think of this like a video game where the AI gets points (rewards) for doing the right thing. For a long time, the standard strategy was to act like a greedy gamer: "Find the single move that gives the most points right now, and do that over and over again." This works well, but it has a flaw. It's like a student who memorizes one specific way to solve a math problem and refuses to try any other method, even if that other method is just as good. They become "stuck" in one way of thinking, missing out on the creativity and variety needed to handle new, tricky situations.

To fix this, scientists recently tried a different approach called "Generative Flow Networks" (GFlowNets). Instead of just hunting for the single best move, this method teaches the AI to explore all the good moves, giving them a chance to be used based on how well they work. It's like telling the student, "Try every valid path through the maze, not just the one that looks fastest." The goal is to create a diverse, flexible thinker. However, trying to scale this up to massive, super-smart AI models has been a nightmare. The tools scientists built to make this work were so complicated and unstable that they often caused the AI to crash or learn nothing at all, especially when the models grew huge or the tasks became messy.

This paper, titled "GFlowRL," tackles that exact headache. The authors discovered that the most complicated part of the previous "diverse thinking" recipe was actually the problem itself. They found that a specific mathematical tool, which was supposed to help balance the AI's learning, was acting more like a noisy, broken radio than a helpful guide. By throwing that broken tool away and replacing it with a much simpler, on-the-fly calculation, they created a new method called GFlowRL. This new approach is stable, fast, and surprisingly effective. It allows massive AI models to learn diverse problem-solving strategies without crashing, even on the hardest math and coding challenges. In fact, their new method outperformed previous record-holders, reaching a skill level in competitive programming that rivals the very best AI systems currently available, all while keeping the training process smooth and steady.

The Problem: The "Magic Calculator" That Broke Everything

To understand the breakthrough, we first need to look at the old way of doing things. When researchers tried to teach AI to be diverse using GFlowNets, they relied on a special "calculator" (technically called a partition function) to help balance the rewards. Imagine you are a teacher trying to grade a class of 1,000 students. You want to make sure that if a student finds a clever, unique solution, they get credit, but you also need to make sure the grades aren't so high that the whole system breaks.

The old method tried to build a brand-new, tiny AI model just to act as this calculator. The problem was that this tiny calculator had to learn from scratch while the giant student (the main AI) was already a genius. It was like asking a toddler to manage the budget for a billion-dollar company while the CEO was making decisions. The toddler (the calculator) would get confused, shout random numbers, and cause the whole system to panic. The researchers found that this "calculator" was actually doing more harm than good. It was so unstable that it caused the AI's learning process to explode with errors, and in many cases, the AI learned nothing better than if the calculator had just been replaced with random noise.

The Solution: Ditch the Calculator, Use the Group

The authors of GFlowRL asked a simple, bold question: "Do we really need this broken calculator?"

They realized that the AI was already doing the work the calculator was supposed to do. When the AI practices, it doesn't just try one answer; it tries a whole group of answers at once (like a focus group). The researchers noticed that they could just look at the group of answers the AI had already generated to figure out the balance they needed. Instead of building a separate, fragile machine to do the math, they just used the data right in front of them.

Think of it like this: Instead of hiring a separate, nervous accountant to tell you how much money the team earned, you just look at the receipts the team just handed you. It's the same information, but it's immediate, it's real, and it doesn't require a new, expensive machine that might break.

This simple switch—replacing the complex, learned calculator with a quick, on-the-spot estimate from the group—changed everything.

Stability: The wild, exploding errors disappeared. The training became as smooth as the standard methods used today.
Simplicity: They didn't need to train a second, extra model. This saved a huge amount of computing power and time.
Performance: Surprisingly, the AI didn't just stop crashing; it got better.

The Results: From Crashing to Champion

The team tested this new method, GFlowRL, on some of the hardest challenges for AI:

Math: They trained models on difficult math competitions. The new method helped the AI solve more problems than any previous method that tried to encourage diversity.
Coding: They tested it on competitive programming sites like Codeforces. A 14-billion-parameter model trained with GFlowRL reached a rating of 2048. To put that in perspective, this is a level of skill that is incredibly high, beating previous open-source records and coming within just 25 points of the very best closed-source AI (o3-mini) available.
Safety (Red Teaming): They also tested it on "red-teaming," which is a way of trying to break AI safety rules to see if they hold up. In this noisy, messy environment where previous methods failed completely, GFlowRL succeeded, achieving the highest success rate in attacking safety filters, proving it could learn complex, diverse strategies even when the feedback was confusing.

Perhaps the most impressive part was how well it scaled. When they tried to use this method on massive "Mixture of Experts" (MoE) models—AI systems with hundreds of billions of parameters—previous methods crashed immediately. GFlowRL, however, kept working perfectly, even on a model with 235 billion parameters.

Why This Matters

The big takeaway here isn't just that they built a better AI trainer; it's that they realized the "best" way to do things isn't always the most complicated way. By stripping away the unnecessary, unstable parts of the old recipe, they found a path that is both simpler and more powerful.

GFlowRL suggests that to make AI truly smart and diverse, we don't need to add more layers of complex machinery. Sometimes, the best way to teach a super-intelligent student is to stop over-engineering the lesson plan and just let them learn from the group of ideas they are already generating. It turns out that the key to scaling up AI reasoning wasn't adding more tools, but knowing exactly which tools to throw away.

Technical Summary: GFlowRL

Problem Statement

Reinforcement Learning (RL) has become the standard post-training paradigm for large language models (LLMs), particularly for reasoning tasks like mathematics and coding. However, dominant RL approaches (e.g., PPO, GRPO) are fundamentally reward-maximizing. They concentrate probability mass on the single highest-reward mode, leading to mode collapse and a reduction in solution diversity. This is structurally limiting for chain-of-thought reasoning, where generalization and robustness rely on covering multiple valid solution paths.

Generative Flow Networks (GFlowNets) offer an alternative by learning policies that sample trajectories in proportion to their reward, thereby matching the reward distribution rather than maximizing it. Recent attempts to apply GFlowNets to LLMs (e.g., FlowRL, FOR) utilize a Trajectory Balance (TB) objective. This objective requires estimating a partition function $Z(x)$ , typically implemented as a learned auxiliary network (e.g., an MLP) conditioned on the prompt.

The paper identifies a critical scaling bottleneck: in the context of modern LLM post-training, jointly training a randomly initialized partition network alongside a massive, pre-trained policy is unstable. The policy requires only modest refinement, while the partition function must learn a complex quantity from scratch within the same short training horizon. This asymmetry causes the learned partition function to act as a source of noise and gradient instability rather than a useful normalizer, often leading to training divergence, especially in sparse-reward regimes (e.g., adversarial red-teaming) and on sparse architectures (Mixture-of-Experts, or MoE).

Methodology: GFlowRL

The authors propose GFlowRL, a streamlined GFlowNet-style RL algorithm that eliminates the learned partition function entirely while preserving the reward-distribution-matching objective.

Core Insight

Through systematic analysis, the authors demonstrate that the learned partition function $Z_\phi(x)$ contributes negligible information to the policy update. Empirically, replacing the learned $Z_\phi$ with random noise yields comparable performance, yet the learned version introduces orders-of-magnitude larger gradient norms and frequent gradient explosions.

The Algorithm

GFlowRL replaces the learned $Z_\phi(x)$ with a simple, in-batch Monte Carlo estimate computed directly from the rollout group already required for training (similar to the group-relative normalization in GRPO).

In-Batch Estimator: For a prompt $x$ and a group of $G$ rollouts $\{y^{(1)}, \dots, y^{(G)}\}$ , the estimate for $\log Z(x)$ is computed as the average of the TB targets across the group:
$\hat{Z}_t(x) := \frac{1}{G} \sum_{i=1}^G \left( \beta r(x, y^{(i)}) + \log \pi_{ref}(y^{(i)} | x) - \log \pi_{old}(y^{(i)} | x) \right)$
This estimator serves as a stop-gradient baseline. It centers the residual but carries no gradient of its own, removing the need for an auxiliary network and its optimizer state.
Stabilizers: To ensure stability, GFlowRL incorporates two key mechanisms:
- Importance-Sampling Correction: Corrects for the drift between the rollout policy ( $\pi_{old}$ ) and the current policy ( $\pi_\theta$ ).
- Asymmetric Flow-Gap Clipping: The flow gap (the residual before the policy update term) is clipped asymmetrically. This allows for more aggressive upward corrections for under-sampled correct solutions while bounding negative updates, preventing outlier rollouts from destabilizing training.
Loss Function: The final loss minimizes the squared residual of the flow gap, normalized by response length to prevent long sequences from dominating the gradient.

Key Contributions

Identification of Instability Source: The paper identifies the learned partition function $Z_\phi$ as the primary source of instability in GFlowNet-based RL for LLMs. It provides empirical evidence that $Z_\phi$ offers almost no benefit for post-training and instead introduces unnecessary optimization challenges.
GFlowRL Algorithm: The proposal of a streamlined algorithm that replaces the learned partition network with an in-batch Monte Carlo estimate. This removes the auxiliary network and its distributed training overhead while retaining the trajectory balance objective's fixed point.
Stable Scaling: Demonstration of stable training across both dense and sparse (MoE) architectures, including regimes where previous methods (FlowRL) diverge.

Experimental Results

The authors evaluate GFlowRL on mathematical reasoning, competitive programming, and adversarial red-teaming benchmarks using models ranging from 7B to 235B parameters.

Mathematical Reasoning: On the 7B dense model, GFlowRL achieves an average accuracy of 40.92, outperforming GRPO (32.48) and FlowRL (35.63). It also surpasses FlowRL on the 32B model.
Code Generation: At the 14B scale, GFlowRL reaches a Codeforces rating of 2048, exceeding DeepCoder-14B (+112 Elo), FlowRL-14B (+144 Elo), and OpenAI's o1 (+157 Elo). It is within 25 Elo of o3-mini.
Adversarial Red-Teaming: In noisy-reward regimes where FlowRL fails to converge, GFlowRL trains stably. It achieves the highest average Attack Success Rate (ASR@1) on AdvBench (82.5%) and HarmBench (79.5%), outperforming the previous SOTA multi-turn attacker (SEMA).
MoE Scaling: GFlowRL successfully trains on Qwen3-30B-A3B (3B active parameters) and Qwen3-235B-A22B (22B active parameters). On the 30B model, it reaches 1999 Codeforces Elo. In contrast, FlowRL diverges on all MoE benchmarks tested.
Diversity: GFlowRL achieves a diversity score of 3.93 (on a 1-5 scale), significantly higher than GRPO (1.21) and FlowRL (2.64), confirming its ability to maintain diverse reasoning paths.

Significance and Claims

The paper claims that GFlowRL is the first GFlowNet-style RL algorithm to scale stably across both dense and sparse architectures for large reasoning models.

The authors argue that the scaling bottleneck for GFlowNet-style RL is not the trajectory-balance objective itself, but the learned partition function. By replacing this component with a simple, non-learned estimator derived from the rollout group, GFlowRL achieves the benefits of distribution matching (diversity, robustness) without the engineering overhead and instability of auxiliary networks. This suggests that scaling GFlowNet-style RL to frontier models depends less on adding complex machinery and more on identifying and removing unnecessary components from the original objective.