💻 computer science

Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learning

This survey presents a modularized framework for Reinforcement Learning in LLMs by categorizing algorithm design into MDP creation, exploration, and learning stages, revealing a significant research bias toward critic-free policy gradients and Monte Carlo methods while highlighting unexplored opportunities in value-based, off-policy, and bootstrapping techniques.

Original authors: Zhao Yang, Yuxuan Jiang, Ting-Chih Chen, Lincen Yang, Annie Wong, Chao Gao, Jacob E. Kooi, Zhong Li, Jiayang Shi, Kevin Qiu, Qi Huang, Xinrui Zu, Shiping Yang, Hengyuan Zhang, Ngai Wong, Filip Ilievsk

Published 2026-06-23

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhao Yang, Yuxuan Jiang, Ting-Chih Chen, Lincen Yang, Annie Wong, Chao Gao, Jacob E. Kooi, Zhong Li, Jiayang Shi, Kevin Qiu, Qi Huang, Xinrui Zu, Shiping Yang, Hengyuan Zhang, Ngai Wong, Filip Ilievski, Shujian Yu, Aske Plaat, Zhaochun Ren, Mark Hoogendoorn, Vincent François-Lavet

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a brilliant but very literal student (the Large Language Model, or LLM) how to write a perfect essay, solve a complex math problem, or write code. You can't just give them a grade on every single word they type as they go. Instead, you wait until they finish the whole thing and then say, "Good job!" or "Try again."

This is the core challenge of Reinforcement Learning (RL) for AI: How do you teach a student when the feedback is delayed, sparse, and only comes at the very end?

This paper is a massive "map" or "survey" that organizes all the different ways researchers are trying to solve this problem. The authors argue that while everyone is using a few popular methods right now (like PPO and GRPO), there is a whole toolbox of other techniques from the world of robotics and game AI that haven't been tried yet. They break the problem down into four main stages, like building a house:

1. Laying the Foundation: Defining the Game (MDP Creation)

Before the AI can learn, you have to define the rules of the game. The paper calls this creating a "Markov Decision Process" (MDP).

The Reward (The Grade): This is the most important part. If you tell the AI "be helpful," how do you measure that?
- The Paper's Insight: Most people use a "Reward Model" (a second AI trained to guess what humans like) or simple rules (e.g., "Did the code run?").
- The Trap: Sometimes the AI gets "smart" in a bad way. If you reward it for writing long answers, it might just write nonsense to get a high score. This is called Reward Hacking. The paper warns that if your rules aren't perfect, the AI will find loopholes.
The State (The Context): The AI needs to know what it has already written. Usually, this is just the text so far. But if the text gets too long, the AI gets overwhelmed. Some researchers are trying to summarize the past or hide parts of it to keep the AI focused.
The Action (The Next Word): Usually, the AI can pick any word from its dictionary. Some researchers are trying to restrict this (like forcing the AI to only pick words that fit a specific grammar) to make learning easier.
The End (Termination): When does the AI stop? Usually, it stops when it types a special "end of sentence" token. But sometimes the AI talks too much. The paper notes that researchers are experimenting with ways to tell the AI, "Stop when you're done," without just cutting it off.

2. The Art of Guessing: Exploration

The AI starts out knowing nothing. It has to try different things to see what works. This is called Exploration.

Temperature (The Dice Roll): Imagine the AI is choosing a word. If you set the "temperature" low, it picks the safest, most obvious word. If you set it high, it takes wilder guesses. This is the simplest way to make the AI try new things.
Entropy (The "Don't Get Bored" Bonus): To stop the AI from getting stuck in a loop of saying the same safe words, researchers add a bonus for being "uncertain" or diverse. It's like telling the student, "I'll give you extra points if you try a different approach, even if it might fail."
Curiosity (Intrinsic Motivation): What if the AI gets a reward just for doing something it hasn't seen before? Some methods give the AI a "curiosity score" for trying new paths, even if it doesn't get a perfect answer yet.
Tree Search (The "What If" Game): Instead of just writing one sentence at a time, imagine the AI branches out like a tree. It writes three different sentences, sees where they lead, and then picks the best path. This is like a chess player thinking three moves ahead.
Curriculum Learning (The Ladder): Don't start with a PhD thesis. Start with a kindergarten story. This method teaches the AI easy problems first, then gradually makes them harder, so it doesn't get discouraged.

3. The Learning Process: How the AI Improves

Once the AI has tried things and gotten feedback, how does it actually learn? The paper categorizes this into four big choices:

Model-Free vs. Model-Based:
- Model-Free: The AI just remembers "I did X, I got a good grade. I'll do X again." It learns by trial and error. This is what most current AI does.
- Model-Based: The AI tries to build a mental map of the world first ("If I say X, Y usually happens"), and then plans based on that map. This is harder but can be more efficient.
Value-Based vs. Policy-Based vs. Actor-Critic:
- Policy-Based: The AI just learns a set of habits (a "policy") to get good grades.
- Value-Based: The AI learns to predict "How good is this situation?" before it even acts.
- Actor-Critic: A team approach. One part (the Actor) decides what to do, and a second part (the Critic) judges how good that decision was. The paper notes that while "Actor-Critic" is the gold standard in robotics, most AI researchers are currently using "Critic-Free" methods because they are cheaper to run on massive computers.
On-Policy vs. Off-Policy:
- On-Policy: The AI learns only from the exact things it just did. If it makes a mistake, it throws that data away and tries again. This is safe but wasteful.
- Off-Policy: The AI learns from its past mistakes and successes, even if it's using a slightly different strategy now. This is like a student reviewing their old test papers to learn from errors. The paper points out that very few AI researchers are doing this yet, even though it's a huge advantage in other fields.
Credit Assignment (Who Gets the Credit?):
- If the AI writes a 100-word essay and gets an "A," which 5 words were the most important?
- Current Default: Most methods just say, "Great job on the whole essay!" and give credit to every word equally.
- The Gap: The paper argues we need better ways to figure out exactly which words led to the success, especially for long, complex reasoning tasks.

4. The Big Picture: What's Missing?

The authors' main conclusion is that the field is unbalanced.

The Crowd: Almost everyone is using the same few tools (Critic-free methods, Monte Carlo credit assignment). It's like everyone in a city driving the same model of car on the same road.
The Empty Lots: There are huge, well-paved roads in the world of Reinforcement Learning (like Off-Policy learning, Value-based methods, and Bootstrapping) that are completely empty in the world of AI.
The Opportunity: The paper suggests that by borrowing these "forgotten" techniques from the broader world of RL, we might be able to teach AI to reason better, learn faster, and handle harder tasks without needing as much computing power.

In short: This paper is a call to stop reinventing the wheel. It says, "We are using a very narrow set of tools to train AI. There is a whole warehouse of other powerful tools that work great in other fields, and we should start trying them out for language models."

Technical Summary: Modularized Reinforcement Learning on LLMs

Problem Statement

Reinforcement Learning (RL) has become central to the post-training of Large Language Models (LLMs), primarily for alignment with human preferences and enhancing reasoning capabilities. However, current pipelines are dominated by a narrow subset of RL methods, specifically Proximal Policy Optimization (PPO) and Group Relative Policy Optimization (GRPO). The existing literature is fragmented, with recent model releases and research papers proposing isolated modifications (e.g., new advantage estimators, clipping strategies, or curriculum schedules) without situating these contributions within a unified framework of RL design decisions. This lack of a shared conceptual framework makes it difficult to distinguish substantive differences between methods, identify unexplored design spaces, or systematically transfer proven classical RL techniques to the LLM domain.

Methodology

This work proposes a principled, modular taxonomy for analyzing and constructing RL algorithms for LLMs. Rather than organizing by application or training stage, the survey structures the examination around three core stages of algorithm construction and an additional category for broader frameworks:

MDP Creation: The survey formalizes LLM generation as a Markov Decision Process (MDP). It analyzes the design choices for:
- Reward Functions: Categorizing sources into learned models (RLHF), rule-based/verifiable rewards (RLVR), and LLM-as-Judge. It further examines reward shaping (e.g., length, format) and the phenomenon of reward hacking.
- State and Action Spaces: Reviewing representations ranging from full token sequences to compressed latent embeddings, and action spaces from full vocabulary to constrained or latent actions.
- Termination and Discounting: Discussing how episode length is controlled (hard cutoffs vs. reward shaping) and the standard practice of setting the discount factor $\gamma=1$ for finite-horizon LLM tasks.
Exploration: The paper categorizes mechanisms used to encourage diverse behavior and escape local optima:
- Stochastic Sampling: Temperature sampling and entropy regularization.
- Intrinsic Motivation: Novelty signals based on token-level or trajectory-level prediction errors, pseudo-counts, and uncertainty.
- Structured Search: Tree search methods (e.g., MCTS) that branch at divergence points.
- Curriculum Learning: Strategies for data sampling, starting state initialization, and reward signal scheduling (dense-to-sparse).
Learning: The survey maps LLM training methods onto four classical RL dimensions:
- Model-Free vs. Model-Based: Noting that LLMs are predominantly trained via model-free updates, though the pre-trained model can be viewed as an implicit dynamics model.
- Solution Representation: Contrasting value-based, policy-based (critic-free), and actor-critic methods. It highlights the dominance of critic-free policy gradients (e.g., GRPO, RLOO) in LLMs, contrasting with the value-based dominance in classical RL.
- Data Source: Analyzing the prevalence of on-policy methods (PPO, GRPO) versus off-policy approaches (e.g., DPO, experience replay variants), and the challenges of distribution mismatch and the "deadly triad" in the LLM context.
- Credit Assignment: Examining how rewards are attributed to tokens, contrasting Monte Carlo (full trajectory) methods with bootstrapping (TD) approaches. It details recent refinements to GRPO regarding advantage estimation, importance sampling, and clipping.
Other Frameworks: The survey briefly covers Meta-RL, Multi-Agent RL (MARL), Offline RL, and Hierarchical RL (HRL) as emerging paradigms in LLM training.

Key Contributions

Unified Taxonomy: The primary contribution is a structured taxonomy that maps the fragmented LLM literature onto a comprehensive RL design space. This allows for a systematic comparison of methods that might otherwise appear unrelated.
Identification of Research Gaps: By mapping current efforts onto this taxonomy, the paper reveals a strikingly non-uniform distribution of research. It identifies that while critic-free policy gradients and Monte Carlo credit assignment are densely populated, other well-established classical RL techniques remain largely unexplored in LLMs. Specific gaps include:
- Value-based methods and off-policy actor-critic training (specifically SAC-style updates).
- Bootstrapping-based credit assignment (moving beyond pure Monte Carlo).
- Systematic integration of discount factors for length control.
- Multi-objective reward composition and open-ended task rewards.
Clarification of Design Trade-offs: The survey elucidates why certain defaults (e.g., $\gamma=1$ , critic-free methods) have become standard in LLMs (e.g., due to memory costs of full-sized critics and the resettable nature of LLM environments) and where these defaults may be suboptimal (e.g., for long-horizon reasoning tasks).

Results and Findings

The paper does not present new experimental results but synthesizes findings from the existing literature to draw structural conclusions:

MDP Formulation: The standard token-level, full-vocabulary, undiscounted MDP is a reasonable default but admits beneficial alternatives (e.g., latent states, constrained actions) in specific settings.
Exploration: Current exploration strategies are fragmented and often operate at the token level rather than the strategy level. There is limited empirical evidence comparing the synergies of different exploration methods.
Learning: The dominance of critic-free methods (like GRPO) is driven by the ability to substitute compute (large-scale sampling) for a learned critic and the memory efficiency gained by avoiding a second full-sized model. However, this comes at the cost of sample efficiency and potential instability in long-horizon tasks where value-based or actor-critic methods could offer benefits.
Credit Assignment: Most methods rely on Monte Carlo updates with trajectory-level advantages. Recent work attempts to refine this (e.g., VinePPO, TEMPO), but fundamental shifts toward bootstrapping or per-step rewards remain an open challenge.

Significance

The paper claims significance by providing a shared conceptual framework for researchers in both RL and LLMs. By making the "structural gaps" explicit, it aims to:

Facilitate Knowledge Transfer: Enable RL researchers to identify which classical techniques (e.g., off-policy actor-critic, bootstrapping) have not yet been applied to LLMs and could be promising candidates for transfer.
Guide LLM Researchers: Help LLM researchers understand the "why" behind current algorithmic choices and identify underexplored directions for improvement beyond incremental refinements of GRPO or PPO.
Reframe the Field: Shift the discourse from isolated algorithmic tweaks to a modular understanding of RL design decisions, highlighting that the current concentration of effort in specific areas is a choice rather than a necessity.

The authors maintain a modest stance, noting that the identified gaps do not necessarily represent "easy wins" and that there are often valid reasons (e.g., computational constraints, architectural mismatches) why certain classical techniques have not been adopted. Nevertheless, they argue that a shared framework is a productive starting point for investigating these structural opportunities.