💬 NLP

Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Space

This paper introduces Natural Language Actor-Critic (NLAC), a scalable off-policy learning algorithm that utilizes a generative LLM critic to provide natural language feedback instead of scalar rewards, thereby improving training stability and sample efficiency for LLM agents in complex, long-horizon tasks.

Original authors: Joey Hong, Kang Liu, Zhan Ling, Jiecao Chen, Sergey Levine

Published 2026-02-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Joey Hong, Kang Liu, Zhan Ling, Jiecao Chen, Sergey Levine

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a very smart but inexperienced robot to navigate a complex maze, solve puzzles, or chat with customers. The robot is powered by a Large Language Model (LLM)—basically, a super-charged text generator that knows a lot of facts but hasn't learned how to act in the real world yet.

The paper introduces a new training method called Natural Language Actor-Critic (NLAC). Here is how it works, explained through simple analogies.

The Problem: The "Silent Scorecard"

Traditionally, when we train these robots, we use a method called Policy Gradient. Imagine the robot tries to solve a puzzle. It makes 20 moves, and at the very end, you give it a single grade: "Pass" or "Fail."

The Issue: If the robot fails, it doesn't know which of the 20 moves caused the failure. It's like getting a "F" on a 20-page essay without any red ink marking the specific mistakes. The robot has to guess what went wrong, which is slow, inefficient, and often leads to the robot getting confused and giving up.

The Old Solution: The "Scalar Critic"

To fix this, researchers tried Actor-Critic methods. They added a "Critic" (a coach) that gives feedback after every move, not just at the end.

The Issue: Traditional critics give a single number (a score from 0 to 1).
The Metaphor: Imagine a coach shouting, "Good job!" or "Bad job!" with a number like "7.5" or "2.1."
- If the robot makes a mistake, a number doesn't tell it why it was a mistake or how to fix it. It's like a teacher saying "You got a 60%" but not explaining that you forgot to carry the one in the math problem. The robot is left guessing how to improve.

The New Solution: NLAC (The "Talkative Coach")

The authors propose NLAC, which replaces the number-crunching coach with a talkative, reasoning coach.

1. The "Language Critic" (The Coach)

Instead of giving a number, the Critic (which is also an LLM) writes a short paragraph explaining the move.

How it works: After the robot makes a move, the Critic says: "That move was okay, but you picked the wrong tool. You should have checked the weather first because the user mentioned rain. If you had done that, you would have saved time."
The Magic: The robot can read this explanation, understand the logic, and learn exactly how to change its behavior. It's like getting a detailed teacher's note instead of just a grade.

2. The "Language Bellman Backup" (The Time Traveler)

The hardest part of training is predicting the future. Usually, to teach a robot, you have to watch it play the whole game 1,000 times to see what happens. This takes forever.

The Innovation: The NLAC Critic is trained to be a Time Traveler. Instead of watching the whole game, it looks at the next step and then uses its imagination to write a short summary of what the rest of the game would look like.
The Metaphor: Imagine you are playing chess. Instead of playing out the whole game to see if a move is good, your coach instantly writes a story: "If you move here, your opponent will likely attack your queen, leading to a loss in 5 moves."
Why it matters: This allows the robot to learn from just one step of experience, rather than waiting for the whole game to finish. It makes learning incredibly fast and efficient.

3. The "Refinement Policy" (The Editor)

Once the Critic writes its feedback, the robot doesn't just memorize it; it edits its own work.

How it works: The robot thinks, "The coach said my move was bad because I didn't check the weather. Let me rewrite my move to check the weather first."
The Result: The robot practices "self-correction" using the coach's written advice, refining its actions until they are perfect.

Why This is a Big Deal

The paper tested this on three types of tasks:

Math Reasoning: Solving hard math problems.
Dialogue Games: Playing "20 Questions" to guess an object.
Customer Service: Handling complex requests like changing flight bookings or returning items.

The Results:

Speed: NLAC learned much faster than previous methods. It needed fewer "tries" to get good at the task.
Stability: It didn't get confused or "break" as easily as older methods.
Performance: On complex, multi-step tasks (like the customer service scenarios), NLAC significantly outperformed other AI training methods, even beating some of the most advanced "frontier" models that were just prompted without any training.

Summary

Think of NLAC as upgrading a robot's training from a silent scoreboard (where you only see the final score) to a personal tutor who:

Predicts the future in a story format (so the robot learns from single steps).
Writes detailed feedback explaining why a move was good or bad.
Guides the robot to rewrite its own actions based on that feedback.

This approach allows AI agents to learn complex, long-term tasks much more efficiently and stably than before.

Technical Summary: Natural Language Actor-Critic (NLAC)

Problem Statement

Large Language Model (LLM) agents are increasingly tasked with complex, long-horizon interactions involving tool-use, web browsing, and multi-turn dialogue. While LLMs excel at single-turn tasks, they struggle with dynamic planning and credit assignment in extended sequences. Traditional reinforcement learning (RL) approaches, such as policy gradients (e.g., PPO, GRPO), often suffer from instability and poor sample efficiency in these settings due to sparse rewards and the difficulty of assigning credit to specific actions within a long trajectory.

Existing Actor-Critic methods attempt to mitigate this by learning a critic to provide denser feedback. However, standard scalar-valued critics are ill-suited for language action spaces; they compress rich failure modes into a single number, offering little guidance on how a language policy should refine its actions. Previous attempts to use natural language values (e.g., Natural Language Reinforcement Learning, NLRL) have been limited by their reliance on in-context aggregation of on-policy rollouts. This approach becomes unscalable as trajectories lengthen and environment dynamics grow complex, restricting such methods to simple environments like gridworlds.

Methodology: Natural Language Actor-Critic (NLAC)

The paper proposes Natural Language Actor-Critic (NLAC), a novel algorithm that trains LLM policies using a generative LLM critic that produces values in natural language space. Unlike prior work, NLAC enables general, scalable, off-policy learning without relying on in-context aggregation of on-policy samples.

Core Components

The framework utilizes a single underlying LLM with different prompts to instantiate three distinct functional components:

Language Successor Model ( $M^\pi$ ):
Instead of predicting scalar returns, the critic is trained to predict compact textual summaries of future rollouts. It acts as a successor function, taking a state $s_t$ and action $a_t$ and generating a distribution over textual descriptions of the future trajectory $(s, a)_{t+1:T}$ and the final reward. This is trained using a Language Bellman Backup ( $B_L$ ), which bootstraps from one-step transitions. The backup combines the immediate next state and action with a bootstrapped description of the future, "discounting" the future description to emphasize immediate outcomes. This allows for off-policy training via temporal-difference learning, circumventing the need for full on-policy trajectories.
Language Evaluator ( $E$ ):
To evaluate a specific action, the critic aggregates multiple sampled future descriptions (generated by the successor model) and produces a textual critique. This critique comments on the optimality of the action and provides a justification based on the predicted future outcomes. This step leverages the LLM's inherent ability to reason over text in-context without additional training.
Refinement Policy ( $\pi_r$ ):
Policy improvement is achieved through a self-refinement paradigm. Rather than enumerating all possible actions (which is intractable in large token vocabularies), the base policy generates an initial action. The refinement policy then takes this action and the textual critique from the evaluator to generate a refined action $a^r_t$ that is expected to be superior according to the critic.

Training Algorithm

The algorithm operates in an off-policy manner:

Policy Evaluation: The language successor model is trained to minimize the divergence between its predicted future descriptions and the target distributions generated by the Language Bellman Backup. This uses a cross-entropy loss (KL-divergence) on the textual outputs.
Policy Improvement: The base policy is updated via distillation from the refinement policy. The objective is to maximize the log-likelihood of the refined actions generated by the refinement policy, conditioned on the state and the critic's evaluation.

Key Contributions

Scalable Off-Policy Learning in Language Space: NLAC is the first work to perform general value learning in natural language space without relying on in-context aggregation of on-policy rollouts. By training a language successor model via a Language Bellman Backup, it enables scalable off-policy learning.
Interpretable and Actionable Critiques: The method replaces scalar values with natural language critiques that explain why an action is suboptimal and suggest improvements. This leverages the LLM's reasoning capabilities to guide policy refinement more effectively than scalar signals.
Theoretical Convergence: The authors provide a theoretical analysis connecting the learned language representations to successor features. They prove that under mild realizability assumptions, the language critic can recover the true scalar Q-function via a monotonic mapping, ensuring convergence to the optimal policy.
Empirical Superiority on Long-Horizon Tasks: The paper demonstrates that NLAC outperforms existing fine-tuning baselines (PPO, GRPO, RFT) and prompting methods (ReAct) on complex tasks requiring multi-step interaction.

Experimental Results

The authors evaluated NLAC on three benchmarks:

Mathematical Reasoning (MATH500-Hard): A single-step task where NLAC performs comparably to other fine-tuning methods, effectively reducing to self-distillation with a generative reward model.
Strategic Dialogue (20 Questions): A multi-turn game requiring cohesive planning. NLAC achieved a 26.0% win rate, significantly outperforming PPO (17.2%), GRPO (18.4%), and the prompting baseline (30.2% with GPT-5, but NLAC with smaller models outperformed fine-tuned baselines).
Tool-Use and Dialogue (τ-Bench): A customer service task involving API calls and strict policy guidelines. NLAC achieved a 0.42 win rate in retail and 0.22 in airline scenarios (unseen during training), outperforming all other fine-tuning baselines. Notably, NLAC required fewer gradient steps to converge compared to PPO.

Qualitative analysis showed that NLAC's natural language critic successfully identified specific failure modes (e.g., violating tool-call limits or using linear search strategies) and provided actionable corrections that scalar rewards could not convey.

Significance and Claims

The paper claims that NLAC offers a more scalable and stable training paradigm for LLM agents compared to existing on-policy methods. By moving value learning from scalar space to natural language space, the method addresses the credit assignment problem in long-horizon tasks more effectively. The authors posit that this approach better leverages the inherent reasoning capabilities of LLMs, allowing them to understand and articulate the "why" behind policy updates.

The authors acknowledge limitations, specifically the reliance on the base LLM's ability to generate plausible futures and refine actions via prompting. They suggest that for prohibitively complex tasks, future work may need to decompose tasks hierarchically to remove this dependency. However, the current results suggest that NLAC is a compelling alternative for training agents in dialogue and tool-use domains.