🤖 AI

When to Plan: Learning to Select Between Reactive Control and Deliberative Planning

This paper introduces a reinforcement learning method for training a meta-reasoning policy that dynamically allocates computation between fast reactive control and slower deliberative planning by using a reactive-policy uncertainty score to predict when planning is necessary, thereby optimizing performance in navigation tasks while adapting as the reactive policy improves.

Original authors: Adam Labiosa, Josiah P. Hanna

Published 2026-07-21

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Adam Labiosa, Josiah P. Hanna

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a puzzle. Sometimes, you can just glance at the pieces and slap them together instantly because you've seen that pattern a thousand times before. That's your brain's "fast lane." But other times, the puzzle is weird, the pieces are missing, or the picture is upside down. In those moments, your fast lane hits a wall, and you have to stop, think hard, and maybe even draw a map before you make a move. That's the "slow lane."

This paper lives in the world of artificial intelligence, specifically a field called meta-reasoning. Think of meta-reasoning as the brain's manager. It doesn't do the actual work; instead, it decides how the work gets done. The big question researchers have been asking is: How do we teach a robot or a computer program to know when to use its fast, instinctive reflexes and when to slow down and do the hard, slow thinking? We care about this because if a robot always thinks slowly, it's too sluggish to catch a ball or dodge a car. But if it always acts fast, it might crash into things it doesn't understand. The goal is to build an agent that knows exactly when to switch gears.

The authors of this paper, Adam Labiosa and Josiah P. Hanna, decided to teach an AI agent this exact skill. They created a "meta-agent" that acts like a traffic cop for the computer's brain. This agent has two main tools in its toolbox: a Reactive Policy and a Planner.

The Reactive Policy is like a reflex. It's a super-fast neural network that looks at the situation and immediately says, "Do this!" It's incredibly quick because it doesn't stop to think. However, it has a blind spot: it only knows how to handle situations it has seen before. If you put it in a new, weird room, it might panic and make a terrible move.

The Planner, on the other hand, is like a cautious architect. It takes time to simulate different paths, look ahead, and figure out the perfect sequence of moves. It's much more reliable in new or tricky situations, but it's slow. Using the planner costs "time," just like spending extra minutes to think before answering a question.

The paper's main discovery is a clever way to teach the meta-agent to choose between these two tools. Instead of guessing, the meta-agent looks at a specific signal: uncertainty. The reactive policy is built as a team of several neural networks working together. If all the networks agree on what to do, the uncertainty is low, and the meta-agent says, "Go ahead, use the fast reflex!" But if the networks start arguing with each other, the uncertainty is high. This tells the meta-agent, "Whoa, this looks weird. The fast reflex might mess up. Let's stop and use the slow planner instead."

The researchers tested this idea in a bunch of different video-game-like worlds, from pushing boxes in a grid to navigating a robot arm through obstacles. They found that their smart, adaptive agent was much better at reaching its goals than agents that were forced to always be fast or always be slow. In fact, in some tricky scenarios, their agent was up to 2.5 times faster than the "always plan" bots and much more successful than the "always react" bots.

One of the coolest findings is that this system is self-improving. The researchers set up a scenario where the "fast reflex" agent gets better over time by learning from the "slow planner." As the reflex agent learned more and became more confident, the meta-agent noticed the uncertainty signals dropping. Consequently, the meta-agent started trusting the reflex more and more, eventually switching to almost entirely fast, reactive control. It's like a student who starts by asking a teacher for help on every math problem, but as they get smarter, they realize they can solve most problems on their own and stop asking for help.

The paper also explored how different factors change the agent's behavior. They found that if the "cost" of thinking (the time it takes to plan) goes up, the agent naturally becomes more lazy and relies on its reflexes more often. They also found that if the environment becomes very chaotic and noisy, the agent stops using long, complex plans because those plans are too likely to get messed up by the noise, and instead sticks to shorter plans or quick reflexes.

In short, this paper suggests that by giving an AI a simple way to measure its own confusion (uncertainty), we can teach it to be the perfect balance of a speedy reflex and a thoughtful planner. It doesn't just work in one specific game; the method seems to adapt to different types of challenges, proving that a little bit of self-awareness goes a long way in making artificial agents smarter and more efficient.

Technical Summary: When to Plan: Learning to Select Between Reactive Control and Deliberative Planning

Problem Statement

Artificial agents often face a tradeoff between reactive control (fast, model-free policies that map states directly to actions) and deliberative planning (slower, model-based decision-time planning that produces higher-quality actions but incurs computational latency). While reactive policies are efficient within their training distribution, they often generalize poorly to out-of-distribution (OOD) states. Conversely, planners are robust across broader state spaces but introduce time delays that can degrade performance in time-sensitive tasks.

Current systems typically commit to a fixed strategy (always reactive or always planning) at test time. This paper addresses the meta-reasoning problem: how to learn an adaptive policy that dynamically selects between a fast, distribution-constrained reactive policy and a slower, reliable planner to maximize task return under a time-based cost. The core challenge is determining when the reactive policy is likely to fail (requiring planning) versus when it is "good enough" (allowing for immediate execution).

Methodology

1. Problem Formulation

The authors formalize the problem using the options framework (Sutton et al., 1999) within a Meta-Markov Decision Process (Meta-MDP):

Meta-State ( $\tilde{S}$ ): The agent's internal state, distinct from the world environment state.
Options ( $O$ ): A set of choices including a reactive option ( $\pi_r$ ) and multiple planning options ( $\pi_k$ ) with varying horizons $k$ .
Cost Model: The objective is to minimize time-to-goal.
- Reactive Option: Incurs no planning cost ( $c_p=0$ ) and executes one step ( $c_a=-1$ ).
- Planning Option: Incurs a cost proportional to search depth ( $c_p(k) = -\alpha k$ ) plus the execution cost of the plan steps.
- Reward: $\tilde{r} = c_p(k) + k \cdot c_a$ .

2. Meta-Policy Architecture

The meta-policy ( $\pi_\Omega$ ) is trained via Reinforcement Learning (PPO) to select options based on an internal observation vector $\tilde{s}_t$ . Crucially, the meta-observation does not include the raw environment state to avoid the same distributional shift problems faced by the reactive policy. Instead, it relies on:

Uncertainty Score ( $\tilde{u}_t$ ): Derived from an ensemble of $M$ $M$ neural networks representing the reactive policy.
- For continuous actions: Variance of the ensemble means.
- For discrete actions: Entropy of the averaged categorical distribution.
- High uncertainty indicates OOD states where planning is likely beneficial.
Distance to Goal ( $\tilde{d}_t$ ): A task feature indicating progress.
Previous Meta-Action ( $\tilde{a}_{t-1}$ ): Context for the last decision.
History: A sequence of the last $n$ observations to track trends in uncertainty and progress.

3. Execution Loop

At each meta-step:

The reactive policy generates an action and an uncertainty score.
The meta-policy constructs the observation $\tilde{s}_t$ and selects an option (reactive or a specific planning horizon).
If reactive, the action is executed for one step.
If planning, a plan of horizon $k$ is computed (using an A* planner with a perfect world model) and executed open-loop for $k$ steps.
The cycle repeats upon option termination.

4. Training Strategy

Reactive Policy: Pre-trained via behavior cloning on in-distribution trajectories generated by the planner.
Meta-Policy: Trained with PPO to maximize discounted return (minimizing time cost).
Joint Training (Optional): In some experiments, the reactive policy is fine-tuned via behavior cloning on planner rollouts during meta-policy training, allowing the system to adapt as the reactive policy improves.

Key Contributions

RL-Based Meta-Reasoning for Computation Allocation: The paper introduces a method to learn a meta-policy that arbitrates between a fast, potentially unreliable reactive policy and a slow, reliable planner. Unlike prior work that selects internal computations or repeats actions, this approach treats the selection between distinct control paradigms as a sequential decision problem across a full task episode.
Environment-Adaptive Strategies: The authors demonstrate that the learned meta-policies successfully balance planning and reacting based on environment characteristics, including planning delay ( $\alpha$ ), reactive policy competence, and environmental stochasticity.
Uncertainty-Conditioned Adaptation: The study shows that conditioning the meta-policy on reactive uncertainty allows it to track improvements in the reactive policy. As the reactive policy learns and its uncertainty decreases, the meta-policy naturally shifts toward fully reactive control, optimizing for speed without sacrificing performance.

Experimental Results

The method was evaluated on five motion planning and navigation environments (Box Push, DoorKey, Fetch, Maze, Navigation) with varying state spaces (discrete/continuous) and task structures.

Performance vs. Baselines: The adaptive meta-policy consistently outperformed fixed-compute baselines (always-reactive, always-short-plan, always-medium-plan, always-long-plan) in terms of total episode return (time cost).
- Box Push: The meta-agent achieved nearly 2x improvement over baselines by avoiding reactive control in states where mistakes are irrecoverable.
- Fetch: Achieved 2.5x improvement, leveraging reactive control for generalizable parts of the task while planning for complex 7-DOF arm maneuvers.
Ablation Studies:
- Uncertainty and History: Removing the uncertainty score or observation history caused significant performance drops (e.g., 4.5x worse in Box Push), confirming these are the most critical components for effective allocation.
- Distance and Previous Action: While helpful, their removal had less impact than uncertainty/history in most domains.
Sensitivity Analysis:
- Planning Cost: As the cost of planning ( $\alpha$ ) increased, the meta-policy shifted toward reactive control.
- Stochasticity: In noisy environments, the meta-policy reduced reliance on long-horizon plans (which compound errors) but maintained a mixed strategy of reactive and short/medium planning.
- In-Distribution Coverage: As the proportion of in-distribution tasks increased, the meta-policy correctly shifted to fully reactive control.
Joint Training: In settings where the reactive policy improved over time, the meta-policy successfully detected the reduced uncertainty and converged to exclusive reactive control, maximizing efficiency.

Significance and Claims

The paper claims to provide a practical mechanism for adaptive computation in artificial agents. Its significance lies in:

Operationalizing Meta-Reasoning: Moving beyond descriptive models of human dual-process thinking (System 1 vs. System 2) to provide a trainable mechanism for AI agents to allocate cognitive resources.
Efficiency without Sacrifice: Demonstrating that agents can achieve near-optimal performance by dynamically switching between fast heuristics and slow planning, rather than committing to a single, suboptimal strategy.
Scalability: The use of uncertainty signals derived from the reactive policy itself allows the system to generalize to OOD states without requiring the meta-policy to understand the raw environment state directly.

The authors note limitations, specifically the assumption of a perfect world model for planning and the use of hand-designed features for the meta-observation. They suggest future work should incorporate learned world models with uncertainty and explore more expressive observation representations for high-dimensional inputs.