💬 NLP

A Pragmatist Robot: Learning to Plan Tasks by Experiencing the Real World

The paper introduces PragmaBot, a framework that enables robots to learn task planning through real-world experience by using a vision-language model for visual self-reflection and a dual memory system (short-term and long-term) to iteratively improve success rates via retrieval-augmented generation.

Original authors: Kaixian Qu, Guowei Lan, René Zurbrügg, Changan Chen, Christopher E. Mower, Haitham Bou-Ammar, Marco Hutter

Published 2026-02-17

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kaixian Qu, Guowei Lan, René Zurbrügg, Changan Chen, Christopher E. Mower, Haitham Bou-Ammar, Marco Hutter

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a brilliant, world-class chef to cook a complex meal for you. This chef has read every cookbook in existence and knows the theory of cooking perfectly. However, there's a catch: this chef has never actually held a knife, felt the weight of a pan, or seen what happens when you try to chop a slippery onion.

If you ask this chef to "chop the onion," they might confidently say, "Easy! Just grab the top and slice!" But in reality, the onion rolls away, or the knife slips because the chef doesn't understand the physical friction of the counter.

This is exactly the problem with current robots using Large Language Models (LLMs). They are smart "brains" that know the words, but they lack the "body" experience to know what actually works in the real world.

PRAGMABOT is the solution to this problem. It's a framework that teaches a robot to learn by doing, failing, and remembering, much like a human child learning to walk.

Here is how it works, broken down into simple concepts:

1. The "Brain" and the "Eyes" (The VLM)

Instead of just reading text, PRAGMABOT uses a Vision-Language Model (VLM). Think of this as a super-smart assistant who can both see the world through the robot's camera and understand language.

The Brain: It plans the steps (e.g., "Pick up the apple").
The Eyes: It watches what happens after the robot moves. Did the apple actually move? Did it fall?

2. The "Short-Term Memory" (The Sticky Note)

When the robot tries a task and fails, it doesn't just give up. It pauses and thinks out loud (a process called Self-Reflection).

The Analogy: Imagine you try to open a jar, but it's stuck. You think, "Okay, that didn't work. Maybe I need to tap the lid first." You write this thought on a sticky note and stick it right in front of you.
In the Robot: This is the Short-Term Memory (STM). If the robot tries to pick up an apple but a can is blocking it, the robot realizes, "I can't reach the apple." It updates its plan on the sticky note: "First, push the can away, then pick up the apple." This allows the robot to fix its mistakes during the same task without needing a human to tell it what to do.

3. The "Long-Term Memory" (The Diary)

Once the robot successfully finishes a task, it takes that sticky note and writes a polished entry in its Diary (the Long-Term Memory or LTM).

The Analogy: You learn that "jars are easier to open if you tap the lid." You write this in your diary. Next time you see a jar, you don't have to figure it out from scratch; you just look at your diary.
In the Robot: The robot summarizes the lesson: "When an object is blocked by a container, push the container first." This lesson is saved forever.

4. The "Librarian" (RAG)

When the robot faces a new task, it doesn't start from zero. It asks its internal Librarian to find relevant pages in its Diary.

The Analogy: You are asked to open a new, weird-shaped jar. You don't guess; you ask your librarian, "Do we have any notes about opening weird jars?" The librarian hands you the page about "tapping the lid."
In the Robot: This is called RAG (Retrieval-Augmented Generation). The robot searches its past experiences to find a similar situation. If it previously learned to push a can out of the way to get an apple, and now it needs to move a screw blocked by a towel, it remembers: "Hey, I pushed a can before; maybe I should push this towel first!"

5. The "Highlighter" (Image Annotation)

Sometimes, the robot's eyes get confused. It might see a drumstick and think, "I should grab the meat," but the meat is slippery and will fall off.

The Analogy: Imagine you are trying to grab a specific part of a complex object, like the handle of a mug. You put a highlighter on the handle to make sure you don't grab the wrong part.
In the Robot: PRAGMABOT can draw "masks" (digital highlighters) on the camera image to tell the robot exactly where to grab or push. This prevents it from grabbing the wrong part of an object.

The Results: From Clumsy to Clever

The paper tested this on real robots doing tricky tasks, like:

Picking up a tennis ball hidden behind a fan.
Moving a tiny candy without knocking over a nearby sponge.
Picking up a bowl that has an apple inside (so you have to move the apple first).

Without PRAGMABOT: The robot failed about 35% of the time because it kept trying the same wrong thing over and over.
With PRAGMABOT: The success rate jumped to 84%.

Even more impressive, when the robot faced a brand new task it had never seen before, it used its "Diary" to succeed 80% of the time on the very first try. It had learned general lessons (like "clear the path first") that applied to new situations.

The Big Picture

PRAGMABOT is a robot that doesn't need to be retrained by engineers every time it makes a mistake. Instead, it acts like a curious human:

It tries something.
It fails and thinks, "Why did that fail?"
It tries again with a new plan.
It writes down the lesson so it never makes that mistake again.

It turns a robot from a rigid machine that follows code into a pragmatic learner that adapts to the messy, unpredictable real world.

1. Problem Statement

Large Language Models (LLMs) and Vision-Language Models (VLMs) have shown promise in robotic task planning by interpreting natural language instructions. However, a critical gap exists:

Misalignment with Embodiment: Models trained on general internet data lack inherent knowledge of specific robot hardware, skill sets, and physical limitations. They often generate plans that are logically sound but physically impossible (e.g., grasping an object blocked by another object).
Lack of Real-World Feedback: Traditional approaches often rely on extensive fine-tuning or dense human supervision to correct these errors.
Static Knowledge: Existing methods struggle to adapt dynamically during a task or retain lessons learned from failures for future, unseen scenarios without retraining.

The core challenge is how to align a general-purpose VLM with a specific robot's capabilities to achieve robust, adaptive task planning in the real world without expensive model retraining.

2. Methodology: PRAGMABOT Framework

PRAGMABOT introduces a framework inspired by Verbal Reinforcement Learning, where the robot learns through self-reflection and memory retrieval rather than weight updates. The system utilizes a VLM (specifically GPT-4o) in three distinct roles: Task Planner, Success Detector, and Experience Summarizer.

The architecture consists of three core mechanisms:

A. Short-Term Memory (STM) & Online Adaptation

Function: Acts as a dynamic log of the current task execution, storing the sequence of actions ( $a_t$ ) and their corresponding feedback ( $r_{t+1}$ ).
Self-Reflection: When an action fails, the VLM analyzes the STM to identify the cause of failure. Instead of updating neural weights, it uses "linguistic gradients" (natural language reasoning) to generate a chain-of-thought correction.
Emergent Behavior: This allows the robot to adapt mid-task. For example, if a direct grasp fails due to occlusion, the VLM might reason, "I need to push the obstructing object first," and execute a new plan. It can also autonomously discover tool use (e.g., using a sponge to push a small candy).

B. Long-Term Memory (LTM) & Experience Summarization

Function: A persistent database that stores lessons learned from successful task completions.
Summarization: Upon task completion, the VLM summarizes the STM into a concise experience entry ( $E$ ) and stores it alongside a scenario key ( $K$ ). The key combines the user instruction and a natural language description of the initial scene.
Retrieval-Augmented Generation (RAG): When facing a new task, the system embeds the current scenario and retrieves the top- $k$ most relevant past experiences from the LTM using cosine similarity. These retrieved experiences are injected into the planning prompt, allowing the robot to leverage prior knowledge for "first-trial" success in similar scenarios.

C. On-Demand Image Annotation

Problem: Standard VLMs often struggle with precise spatial reasoning for complex objects (e.g., grasping the handle of a mug vs. the body, or pushing a specific point).
Solution: The system employs an on-demand annotation module. If the VLM determines high precision is needed, it triggers Grounded SAM to generate object masks. It then overlays candidate locations (using Farthest Point Sampling for placing/pushing or grasp hypotheses for grasping) onto the image. The VLM selects the optimal location, significantly improving spatial accuracy.

3. Key Contributions

PRAGMABOT Framework: The first system to integrate VLMs with visual feedback loops for real-world robotic planning, enabling learning via STM (online adaptation) and LTM (lifelong learning) without fine-tuning the underlying model.
RAG for Embodied AI: Successfully adapts Retrieval-Augmented Generation to robotic task planning, demonstrating that retrieving self-reflective experiences significantly outperforms naive prompting.
Enhanced Spatial Reasoning: Introduces an on-demand image annotation tool that bridges the gap between high-level semantic planning and low-level 3D action execution, supporting complex skills beyond simple grasping.
Emergent Tool Use: Demonstrates the robot's ability to autonomously identify and use unmentioned objects as tools to overcome physical constraints.

4. Experimental Results

The system was evaluated on a legged manipulator (ANYmal with a 6-DoF arm) across 12 real-world scenarios, including 8 previously unseen tasks.

Short-Term Memory Impact:
- STM-based self-reflection increased task success rates from 35% to 84% on challenging manipulation tasks.
- Enabled the robot to recover from failures by generating new strategies (e.g., pushing obstacles, using tools).
Long-Term Memory & Generalization:
- In single-trial tests on new but similar tasks, PRAGMABOT improved success rates from 22% to 80% compared to baselines without LTM.
- The RAG-based approach achieved 89% accuracy in planning the first action, significantly outperforming random retrieval (17%) and providing the entire LTM context (74%), which often introduced noise.
Image Annotation:
- For complex objects (e.g., drumsticks, skewers), image annotation increased grasp success rates significantly.
- It reduced distance errors in pushing tasks, proving its utility beyond grasping.
Failure Analysis:
- Most failures were attributed to execution issues (poor grasp generation, depth estimation errors) rather than planning logic.
- VLM reasoning errors occurred when the model incorrectly downweighted retrieved memories due to perceived visual misalignment.

5. Significance and Future Work

Paradigm Shift: PRAGMABOT demonstrates that robots can achieve robust, adaptive planning by "learning by experiencing" rather than relying on massive datasets or human-in-the-loop corrections.
Efficiency: It offers a cost-effective solution for deploying robots in dynamic environments, as it avoids the computational and data costs of fine-tuning LLMs.
Generalizability: The framework is hardware-agnostic regarding the VLM and can be applied to various robotic embodiments.

Limitations & Future Directions:

Modality: The system currently relies heavily on vision; integrating tactile or auditory feedback could improve handling of complex interactions.
Memory Scaling: As the LTM grows, strategies for memory pruning (deciding what to forget) and more sophisticated retrieval (e.g., Maximum Marginal Relevance) are needed.
Multi-Robot Learning: Future work aims to explore sharing memories across robots with different morphologies, allowing them to transfer knowledge while filtering out experiences incompatible with their specific hardware.