💻 computer science

Partially Observable Markov Decision Processes (POMDPs) and Robotics

This paper reviews the Partially Observable Markov Decision Process (POMDP) framework for robotics planning, highlighting how recent advances in sampling-based approximate solvers have overcome its historical computational barriers to enable practical, robust applications on physical robots.

Original authors: Hanna Kurniawati

Published 2026-06-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Hanna Kurniawati

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The Robot's Dilemma

Imagine you are driving a car in a thick fog. You can't see the road clearly (partial observability), and your steering wheel might be slightly sticky or your brakes might respond a little differently than expected (non-deterministic effects). You need to get to a destination, but you don't know exactly where you are, and you don't know exactly what your car will do when you turn the wheel.

This is the daily life of a robot. The paper explains how POMDPs (Partially Observable Markov Decision Processes) are the mathematical "brain" designed to help robots make good decisions in this foggy, uncertain world.

The Problem: The "Perfect" Brain is Too Slow

For a long time, mathematicians knew how to build the perfect brain for this situation. This perfect brain would calculate every single possible future, every possible mistake, and every possible outcome to find the one single best move.

However, the paper explains that this "perfect brain" is like trying to solve a puzzle that has more pieces than there are atoms in the universe. It is so computationally heavy that it takes hours or days to figure out a move for a simple problem. For a robot that needs to move in real-time, this is useless. It's like trying to calculate the perfect route for a road trip while you are already stuck in traffic; by the time you finish the math, you've already crashed.

The Solution: The "Good Enough" Explorer

The paper highlights a major breakthrough that happened since the early 2000s. Instead of trying to be perfect, researchers developed sampling-based solvers.

Think of this like exploring a massive, dark cave.

The Old Way (Perfect Solver): You try to map every single inch of the cave, every rock, and every shadow before you take a single step. You never leave the entrance because the map is too big.
The New Way (Sampling Solver): You shine a flashlight. You don't map the whole cave. Instead, you take a few steps, look around, and ask, "If I go left, what's likely to happen? If I go right, what's likely to happen?" You only explore the paths that seem promising. You ignore the dead ends you've already seen.

This approach doesn't guarantee the absolute best path, but it finds a very good path very quickly. This is what makes robots practical today. They can handle uncertainty without freezing up.

The Five Big Hurdles (and How They Were Jumped)

The paper details five specific "monsters" that made POMDPs impossible for robots, and how the new "sampling" methods tamed them:

The Curse of Dimensionality (Too Many Places):
- The Problem: If a robot has 100 possible places it could be, the math explodes. It's like trying to remember every possible combination of a 100-digit lock.
- The Fix: Instead of remembering every number, the robot only remembers the numbers it is likely to encounter. It focuses its memory on the "neighborhoods" it actually visits.
The Curse of History (Too Many Steps):
- The Problem: To make a good decision, a robot needs to think far into the future. But if it thinks 30 steps ahead, the number of possible futures grows exponentially (like a tree branching out wildly).
- The Fix: The robot uses "macro-actions." Instead of thinking about every tiny muscle twitch, it thinks in terms of big goals, like "Go to the kitchen" or "Pick up the cup." This shortens the mental timeline.
The Flood of Data (Too Many Observations):
- The Problem: Robots have cameras, lasers, and sensors. They see millions of pixels. Trying to categorize every single pixel is impossible.
- The Fix: The robot learns to group similar things together. It doesn't care if a wall is pixel #405 or #406; it just cares that "there is a wall." It simplifies the view.
The Infinite Choices (Too Many Actions):
- The Problem: If a robot can move its arm in a continuous smooth motion, there are infinite ways to move it. You can't check them all.
- The Fix: The robot samples a few random movements, tests which ones look promising, and then zooms in on those. It's like tasting a few flavors of ice cream to find the best one, rather than tasting every flavor in the world.
The Complex Physics (Hard to Predict):
- The Problem: Some robots (like race cars or screwdrivers) have complex physics where a small change leads to a huge, unpredictable result. Simulating one step takes a long time.
- The Fix: The robot uses "lazy" simulations. It runs a quick, rough guess first. Only if that guess looks interesting does it run the expensive, detailed simulation.

Real-World Proof

The paper isn't just theory. It mentions that these methods have been put into actual software (like tools called SARSOP, POMCP, and ABT) and tested on real robots.

The Result: In a real-world demo at a robotics conference (ICRA 2018), a robot using these "good enough" POMDP strategies succeeded 100% of the time.
The Comparison: When the same robot tried to do the task without accounting for uncertainty (ignoring the fog), it only succeeded 35% of the time.

The Takeaway

The paper concludes that while we still can't build the "perfect" robot brain that knows everything, we have built a "smart enough" brain that knows how to handle the unknown. By using smart sampling techniques, robots can now navigate uncertainty, gather information, and complete tasks robustly, even when they can't see the whole picture.

In short: We stopped trying to calculate the entire universe and started taking smart, educated guesses. That shift is what made modern, reliable robots possible.

Technical Summary: Partially Observable Markov Decision Processes (POMDPs) and Robotics

1. Problem Definition

The paper addresses the critical challenge of planning under uncertainty in robotics. Autonomous robots must make strategic decisions despite non-deterministic effects of actions (due to actuator errors) and partial observability of states (due to sensor noise and perception limitations). In such scenarios, a robot never knows its exact state but must infer it from a history of actions and observations.

The Partially Observable Markov Decision Process (POMDP) is identified as the mathematically principled framework for modeling these problems. A POMDP is defined by the tuple $\langle S, A, O, T, Z, R \rangle$ , representing state space, action space, observation space, transition function, observation function, and reward function, respectively. The agent maintains a belief $b$ (a probability distribution over states) rather than a single state. The objective is to find an optimal policy $\pi^*: B \to A$ that maximizes the expected total discounted reward.

The Core Challenge: While theoretically sound, computing the exact optimal solution for a POMDP is PSPACE-hard. Historically, this computational intractability rendered POMDPs impractical for robotics, as benchmark problems were limited to fewer than 30 states and required hours to solve, whereas real-world robotics demands real-time performance and scalability to thousands of states.

2. Methodology: Sampling-Based Approximate Solvers

The paper reviews the shift from exact solvers to sampling-based approximate solvers, which have made POMDPs practical for robotics. These methods relax the requirement for global optimality, focusing instead on computing "good" policies within reasonable computational resources.

2.1 General Approach

Sampling-based solvers restrict the problem to a set of representative beliefs rather than the entire belief space. They typically follow a program skeleton involving:

Sampling: Selecting a set of beliefs (often starting from an initial belief $b_0$ ).
Estimation: Estimating the value of these sampled beliefs using heuristics and backup operations.
Update: Iteratively updating the policy $\pi$ based on these estimates.
Termination: Stopping when a time limit is reached or a convergence threshold is met.

These solvers are categorized as Offline (computing a policy before execution) or Online (interleaving policy computation with execution).

2.2 Addressing Specific Computational Barriers

The paper details how specific sampling techniques alleviate five major hurdles:

Large State Space (Curse of Dimensionality):
- Point-Based Value Iteration (PBVI): Samples beliefs reachable from $b_0$ and performs backups only on these points, representing the value function via $\alpha$ -vectors.
- Perseus: Improves efficiency by performing backups on a subset of sampled beliefs.
- HSVI2 & SARSOP: Use upper and lower bounds to guide sampling. SARSOP specifically samples from the set of beliefs reachable under the optimal policy ( $R^*$ ), pruning sub-optimal branches to focus computational effort on relevant regions.
- Continuous Spaces: Methods like MCVI and GCS extend these concepts to continuous spaces using policy graphs or cluster sampling.
Long Planning Horizon (Curse of History):
- Macro-actions: Constructing temporally extended sequences of actions (e.g., via POSMDPs) to reduce the effective depth of the planning tree.
- Automatic Generation: Techniques that automatically generate sub-problems or milestones based on reward density or value of information.
Large Observation Space:
- Discretization: Using value-function-derived features to discretize continuous observations.
- Progressive Widening: POMCPOW uses Double Progressive Widening to incrementally expand the set of observations considered.
- Lazy Belief Extraction: LABECOP avoids discretization entirely by maintaining sampled episodes and reweighting them during execution.
Large Action Space:
- Sampling Actions: Replacing maximization over all actions with maximization over a random subset (e.g., Perseus, GCS).
- Optimization: Using Generalized Pattern Search (GPS-ABT) or Bayesian Optimization for continuous actions, though scalability remains limited for high dimensions.
- Quantile Statistics: Two-stage sampling mechanisms for large discrete action spaces.
Complex Dynamics:
- Hybrid Models: Representing non-linear dynamics as switching state-space models.
- Multi-Level Monte Carlo (MLMC): Approximating expensive dynamic simulations with varying levels of fidelity, using high-fidelity models only occasionally.

3. Key Contributions and Results

The paper does not present new experimental data but synthesizes the state of the art to demonstrate the feasibility of POMDPs in robotics.

Scalability Advances: The review highlights that sampling-based solvers have progressed from handling problems with <30 states to solving benchmarks with 15,000+ states and 1,000+ observations (e.g., HSVI2, SARSOP). SARSOP, for instance, solved the RockSample(10,10) benchmark (over 100,000 states) in seconds.
Practical Application: The paper cites a demonstration at SIMPAR 2018 and ICRA 2018 where a physical robot operated for 7 consecutive days using POMDP strategies. Crucially, even with rough, data-learned transition and observation models, the POMDP approach achieved a 100% success rate, compared to 35% for strategies ignoring uncertainty.
Software Ecosystem: The paper catalogs open-source tools (e.g., Symbolic Perseus, ZMDP, APPL, TAPIR, OPPT, pomdp_py) that implement these solvers and provide interfaces to robotics middleware like ROS, facilitating practical adoption.

4. Significance and Claims

The paper argues that the significance of these advances lies in transforming POMDPs from a theoretical concept applicable only to "toy problems" into a practical tool for realistic robotics.

Robustness: The primary gain of formulating problems as POMDPs is robustness. By explicitly quantifying uncertainty, POMDPs automatically balance the trade-off between information gathering and task execution. This is particularly vital in constrained environments or when operating near the limits of a robot's capability.
Feasibility of Models: The paper claims that accurate models are not strictly necessary; even rough estimates learned from small datasets can yield highly effective policies, provided the solver is scalable enough to handle the uncertainty.
Parallels with Motion Planning: The progression of POMDP solvers mirrors that of sampling-based motion planning (e.g., moving from global planning to local, query-based planning), suggesting that similar theoretical and algorithmic maturation has occurred.
Relation to Learning: The paper notes the close relationship between POMDPs and Reinforcement Learning (RL), particularly Model-Based Bayesian RL, where POMDPs can be used to learn model parameters while solving the task.

In conclusion, the paper posits that while scalability challenges remain (particularly in high-dimensional continuous action spaces), current sampling-based methods are sufficiently efficient to significantly improve the robustness of robotic systems, making POMDPs a viable and recommended approach for planning under uncertainty.