🤖 AI

When in Doubt, Plan It Out: Committed Small Language Model Deliberation for Reactive Reinforcement Learning

The paper introduces PACT, a hybrid architecture that enhances reactive Reinforcement Learning policies by asynchronously integrating a deliberative Small Language Model to generate and validate safe action plans, thereby outperforming baseline methods in challenging environments without requiring policy retraining.

Original authors: Nathan Gavenski, Juarez Monteiro, Francisco Galuppo, Adriano Veloso, Odinaldo Rodrigues

Published 2026-06-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Nathan Gavenski, Juarez Monteiro, Francisco Galuppo, Adriano Veloso, Odinaldo Rodrigues

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are driving a car. Most of the time, you drive on "autopilot." You see a stop sign, you hit the brakes. You see a green light, you go. This is fast, automatic, and requires very little thinking. In the world of AI, this is called Reinforcement Learning (RL). It's great when you're on a familiar road, but if you suddenly find yourself in a completely new city with strange traffic rules, your autopilot might crash because it hasn't seen that situation before.

On the other hand, imagine a very wise, slow-thinking navigator sitting in the passenger seat. This navigator has read every map in the world and can plan a route for a complex journey. However, this navigator is slow to speak, takes a long time to think, and sometimes gets distracted. In the paper, this is the Small Language Model (SLM).

The paper introduces a new system called PACT (Plan, Align, Commit, Think) that combines these two drivers into one perfect team. Here is how it works, using simple analogies:

The Problem: The "Familiar Road" Trap

Standard AI agents are like the autopilot driver. They are excellent at reacting to things they have practiced. But if they encounter something new (like a slippery road or a huge maze), they panic and make mistakes. They lack the ability to "think ahead."

The Solution: The PACT Team

PACT creates a hybrid team with two distinct roles:

The Fast Driver (The RL Policy): This is the autopilot. It handles 90% of the driving. It's quick, efficient, and knows the local rules.
The Slow Navigator (The SLM): This is the planner. It doesn't drive the car. Instead, it sits back and only speaks up when the Fast Driver is confused.

How PACT Works: The "Doubt" Trigger

The system has a simple rule: "When in doubt, plan it out."

The Trigger: The Fast Driver constantly checks its own confidence. If it sees something familiar, it keeps driving. But if it feels "uncertain" (like seeing a slippery patch or a giant maze it hasn't seen before), it hits the brakes and says, "I'm not sure what to do next."
The Planning Phase: When the Fast Driver stops, the Slow Navigator wakes up. It doesn't just say "Turn left." It creates a full roadmap (a plan) for the next few steps.
The Safety Check (Simulation): Before the navigator's plan is used, the system runs a "mental simulation." It asks: If we follow this plan, will we crash? Is it safe? Will it actually get us to the goal? If the plan is risky, the navigator tries again until it finds a safe route.
The Commitment: Once a safe plan is verified, the Fast Driver commits to it. The system ignores the autopilot for a while and follows the navigator's roadmap step-by-step. Even if the road gets a little bumpy (stochasticity), the team sticks to the plan rather than panicking and changing direction every second.
The Alignment: If the car gets slightly off course (maybe the road was slippery and the car slid), the navigator quickly adjusts the route to get the car back on the planned path, rather than starting over from scratch.

The Results: Why It Wins

The researchers tested this team in three different "driving scenarios" (called FrozenLake environments):

The Easy Road (6x6 map): The Fast Driver was good, but the PACT team was even better.
The Slippery Road (6x6 with ice): This is where other teams failed. The Fast Driver slipped and crashed. Other AI systems that asked the navigator for advice every single step got confused and lost. But PACT, because it committed to a verified plan, stayed steady. It only slipped a little bit and recovered.
The Giant Maze (8x8 map): This was a long, complex journey. The Fast Driver got lost. The navigator alone was too slow and wandered aimlessly. But PACT combined them: the navigator drew a clear path, and the Fast Driver executed it perfectly. PACT achieved a perfect score with zero mistakes, while everyone else struggled.

The Big Takeaway

The paper argues that you don't need a super-computer (a massive AI model) to solve hard problems. You just need the right teamwork.

Don't ask the navigator for every turn: That's too slow and confusing.
Don't let the driver guess in the dark: That leads to crashes.
Do this: Let the driver handle the easy stuff. When things get weird, pause, have the navigator draw a safe, verified map, and then stick to that map until the job is done.

In short, PACT proves that a small, smart planner working together with a fast, reactive driver is much more powerful than either one trying to do the job alone.

Technical Summary: Plan, Align, Commit, Think (PACT)

Problem Statement

Reinforcement Learning (RL) agents trained with standard model-free policies excel in familiar environments but degrade significantly when deployed in unfamiliar conditions or novel contexts. This limitation stems from their purely reactive nature: they map states to actions without explicit deliberation or lookahead. While Large Language Models (LMs) offer transferable knowledge and commonsense reasoning suitable for high-level planning, they face two critical barriers in this domain:

Reliability: Even frontier LMs struggle with consistent multi-step planning and state tracking on their own.
Efficiency: Large models are computationally expensive, making them impractical for low-latency, real-time decision-making.

Existing hybrid approaches often treat LMs as step-level advisors (querying the model at every uncertain step) or require manual engineering of planning models (as in BDI systems), failing to bridge the gap between reactive execution and long-horizon planning without costly retraining.

Methodology: The PACT Architecture

The authors propose Plan, Align, Commit, Think (PACT), a hybrid architecture that integrates a fast, reactive RL policy with a slow, deliberative Small Language Model (SLM) planner. Drawing on Kahneman's dual-process theory and Belief-Desire-Intention (BDI) systems, PACT separates decision-making into two components:

Fast Component (Reactive RL): A pre-trained policy (e.g., PPO) handles decision-making in familiar conditions. It operates continuously unless triggered otherwise.
Slow Component (Deliberative SLM): An asynchronous SLM (specifically a 2B-parameter model) is invoked only when the RL policy's epistemic uncertainty exceeds a predefined threshold ( $\tau$ ).

The PACT process consists of three distinct modules:

Plan Generation: Upon high uncertainty, the SLM is prompted with the task description and current observation to generate a candidate action sequence ( $\rho$ ) of up to $n$ steps. This generation relies on a hand-crafted or learned transition function to simulate future states within the prompt.
Plan Verification: The candidate plan is simulated to ensure it is safe (no constraint violations), feasible (within action space), and complete (reaches the goal). If a violation occurs, the SLM is prompted to regenerate a safe plan.
Agent Alignment: This module bridges the gap between the idealized simulation used for planning and the agent's actual current state. If the agent has drifted from the plan due to stochastic transitions, the SLM is prompted to generate a short trajectory to guide the agent back to the nearest reachable waypoint in the verified plan.

Execution Strategy: Once a plan is verified and aligned, the system commits to it. The reactive RL policy is bypassed, and the agent executes the actions directly from the plan until the plan is completed, the goal is reached, or a failure occurs. This commitment mechanism distinguishes PACT from step-level advisory approaches, which lack a unified trajectory to recover from deviations.

Key Contributions

Hybrid Architecture: A novel framework that combines reactive RL with deliberative SLM planning without requiring retraining of the RL policy.
Commitment Mechanism: Unlike step-level advisors, PACT commits to a verified, structured trajectory, allowing the agent to recover from stochastic deviations by re-aligning to the plan rather than re-querying the model at every step.
Efficiency with Small Models: Demonstrates that computationally efficient 2B-parameter SLMs, when wrapped in a verification and commitment loop, can outperform larger models or standalone RL policies in complex, uncertain environments.
Uncertainty-Triggered Planning: The system invokes deliberation only when the RL policy is epistemically uncertain, optimizing the trade-off between speed and planning depth.

Experimental Results

The authors evaluated PACT on the FrozenLake environment across three configurations of increasing difficulty:

6×6 (Deterministic): PACT achieved a reward of 0.98 ± 0.14, outperforming PPO (0.93), standalone SLM (0.47), and other LM-RL baselines (SAYCAN: 0.88, ASK: 0.89).
6×6 (Stochastic/Slippery): PACT maintained high performance (0.93 ± 0.26), showing robustness to stochasticity. In contrast, PPO dropped to 0.64, SAYCAN to 0.53, and ASK to 0.63. PACT required an average of 2.85 replans per episode due to stochastic deviations but maintained a high success rate.
8×8 (Long Horizon): PACT achieved a perfect 1.00 ± 0.00 reward with zero variance. PPO dropped to 0.79, SAYCAN to 0.62, and ASK to 0.74. The standalone SLM performed poorly (0.42) with high variance, indicating directionless wandering without verification.

Key Findings:

LM Usage vs. Performance: PACT's performance is not correlated with the frequency of LM queries but with the structure and commitment to the generated plans. For instance, SAYCAN queried the LM at a similar rate (59.7%) in the stochastic setting but achieved significantly lower rewards (0.53) because it lacked plan commitment.
Scalability: PACT's LM usage increased with task difficulty (27.9% in deterministic, 58.4% in stochastic, 81.2% in long-horizon), indicating the system dynamically allocates deliberative resources based on the environment's demands.

Significance and Claims

The paper claims that deliberative planning and reactive execution are more powerful in concert than either is alone. Specifically, PACT demonstrates that:

Commitment is critical: The ability to commit to a verified plan and re-align to it, rather than treating the LLM as a step-by-step advisor, provides robustness against stochasticity and long horizons that other methods lack.
Verification relaxes scale requirements: By wrapping the SLM in a simulation-grounded verification loop, the system does not require massive parameter counts to achieve reliable planning; a 2B-parameter model suffices.
No Retraining Required: The architecture allows for the integration of pre-trained agents with LLMs without the need for costly retraining or fine-tuning of the RL policy.

Limitations:
The authors acknowledge that PACT currently relies on a hand-crafted or learned transition function for plan simulation, limiting applicability to environments where such dynamics are available. Furthermore, the evaluation was restricted to direct-goal tasks, and the system's performance in tasks requiring complex sub-goal decomposition (e.g., "Door Key" environments) remains a subject for future empirical validation.