🤖 AI

Curiosity-Diffuser: Curiosity Guide Diffusion Models for Reliability

This paper introduces Curiosity-Diffuser, a method that enhances the reliability of robotic imitation policies by using a Random Network Distillation (RND) module to guide conditional diffusion models toward generating trajectories with lower curiosity, thereby reducing hallucinations and overgeneralization while aligning behaviors more closely with the training data.

Original authors: Zihao Liu, Xing Liu, Yuhang Dong, Haitao Chang, Zhengxiong Liu, Panfeng Huang

Published 2026-08-06

📖 8 min read🧠 Deep dive

CC BY 4.0

Original authors: Zihao Liu, Xing Liu, Yuhang Dong, Haitao Chang, Zhengxiong Liu, Panfeng Huang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to do a new trick, like juggling or opening a door. You show it a video of a human doing it perfectly, and the robot's brain (a neural network) tries to copy those moves. This is called "imitation learning." It's like a student memorizing a textbook to pass a test. But here's the catch: what happens when the test question is slightly different from anything in the book? In the real world, things are messy. A robot might encounter a situation it has never seen before, and its brain might start to "hallucinate." Instead of figuring out a safe new move, it might invent a crazy, impossible action that causes it to crash or drop the object. This instability is a huge problem if we want robots to work safely in our homes or hospitals. Scientists have been trying to build robots that don't just memorize, but also know when they are guessing and should be careful.

This paper introduces a clever new way to stop robots from making up dangerous moves. The authors, working with a type of AI called a "diffusion model" (which is like a smart artist that can draw a picture by slowly turning random noise into a clear image), created a system they call Curiosity-Diffuser. Think of the robot's brain as a student who has studied a specific set of practice problems. When the student faces a new problem, they might panic and guess wildly. The Curiosity-Diffuser adds a "curiosity meter" to the robot. If the robot tries to do something that feels too strange or unfamiliar compared to what it learned, the meter goes off. The system then gently steers the robot back toward the safe, familiar moves it knows work, preventing it from wandering into dangerous territory.

The researchers tested this idea in two ways: first in computer simulations where robots had to walk or navigate mazes, and second in the real world with actual robot arms. They found that by using this "curiosity guide," the robots became much more reliable. In the simulations, the robots made fewer mistakes and stayed closer to the safe paths they had practiced. In the real world, when asked to close a cabinet door or stack blocks, the new method helped the robot succeed 85% of the time, compared to only about 50% for older methods. The paper suggests that by teaching the robot to recognize when it is "out of its depth" and guiding it back to safety, we can make intelligent machines that are much less likely to cause accidents.

The Story of the Curious Robot

The Problem: The Robot's Daydream
Imagine you are a robot named Robo. You have been trained by watching a human open a door 100 times. You know exactly how to grab the handle and push. But one day, the door is slightly heavier, or your hand is in a slightly different spot. Your brain, which is really good at copying what it has seen, starts to panic. It doesn't know what to do, so it starts "hallucinating." It might decide to push the door with its elbow, or try to jump over it, or just spin in circles. These are "hallucinations"—invented actions that look like they could work but are actually nonsense. In the real world, this is dangerous. If a robot hallucinates while holding a heavy tool, it could hurt someone.

The Solution: The Curiosity Compass
The authors of this paper realized that robots need a way to check their own confidence. They borrowed an idea from a field called "reinforcement learning" called Random Network Distillation (RND). Let's break that down with a simple analogy.

Imagine you have a "Teacher" and a "Student."

The Teacher is a random, fixed brain that doesn't learn anything. It just looks at a situation and gives a random answer.
The Student is a brain that does learn. It tries to guess what the Teacher would say if it looked at the same situation.

When the Student sees a situation it has practiced a lot (like opening the door), it can predict the Teacher's answer perfectly. The difference between them is tiny. But when the Student sees a weird, new situation (like the heavy door), it gets confused. Its guess is totally different from the Teacher's random answer. This big difference is the "Curiosity" score. High curiosity means "I don't know this!" Low curiosity means "I've seen this before; I'm safe."

How Curiosity-Diffuser Works
The paper combines this Curiosity Compass with a powerful tool called a Diffusion Model. Usually, a diffusion model is like a sculptor who starts with a block of noise and chips away until a statue appears. In robotics, the "statue" is a plan for how to move. The robot generates many possible plans, starting from chaos and slowly refining them.

The authors added a twist: they told the diffusion model, "While you are chipping away the noise to make a plan, keep an eye on your Curiosity Compass."

If the plan starts to look like something the robot has never seen (high curiosity), the compass pushes the plan back toward the familiar.
If the plan looks safe and familiar (low curiosity), the compass lets it continue.

This is like having a coach standing next to the sculptor. If the sculptor starts carving a weird, impossible shape, the coach says, "Whoa, that looks nothing like the practice photos. Let's try a shape closer to the original."

The Results: Safer, Smarter Robots
The team tested this in two main ways:

In the Computer World (Simulations): They used virtual robots that had to walk across a floor or navigate a maze.
- In the MuJoCo tasks (virtual walking robots), the new method helped the robots perform better, especially when the training data was high-quality.
- In the AntMaze tasks (a virtual ant navigating a complex maze), the results were even more impressive. Without the curiosity guide, other robots failed almost completely, getting stuck or wandering off. The Curiosity-Diffuser helped the virtual ant find the goal much more often. For example, in the "Play" group of the AntMaze, the new method succeeded 58.6% of the time, while other methods barely got above 8%.
In the Real World: They took a real robot arm and taught it two tasks:
- Hammer Close Door: Using a hammer to push a cabinet door shut.
- Block Ranking: Picking up a red block and stacking it on a green one.
- They tested the robot with slight changes, like moving the starting position of the arm or the objects.
- The old methods (like standard imitation learning) only succeeded about 50% of the time.
- The new Curiosity-Diffuser succeeded 85% of the time. It was much more stable and didn't get stuck or make wild, unsafe moves.

The "K-Sim" Score: Measuring Safety
The authors also invented a new way to measure how "safe" a robot's moves are. They call it K-Sim.

Imagine you have a map of all the safe places the robot has ever been.
When the robot moves, K-Sim checks: "How close is this new move to the safe map?"
A score close to 1.0 means the robot is staying right on the safe path.
A score close to 0 means the robot is wandering into unknown, dangerous territory.
The Curiosity-Diffuser consistently got higher K-Sim scores (around 0.827 on average in the maze tests) compared to other methods, proving it really was keeping the robot on familiar ground.

A Word of Caution
The paper is careful to say that this isn't a magic wand that fixes everything. The "curiosity weight" (how much the robot listens to the compass) has to be just right. If the robot listens too much to the compass, it becomes too scared to try anything new and might fail to finish the task. If it listens too little, it might still hallucinate. The authors found a "sweet spot" where the robot balances being safe and being effective.

Why This Matters
This research suggests that by giving robots a way to measure their own "familiarity" with a situation, we can stop them from making up dangerous moves. It's a step toward building robots that are not just smart, but also humble and careful—knowing when they are in over their heads and sticking to what they know works. While the paper shows great results in simulations and real-world tests, the authors note that there is still work to do to make these systems even more robust for every possible situation in our complex world.

Technical Summary: Curiosity-Diffuser

Problem Statement
The paper addresses the critical issue of instability and unreliability in robotic imitation learning, particularly when policies encounter out-of-distribution (OOD) states. While recent advancements in deep neural networks and large-scale datasets have enabled sophisticated imitation learning frameworks (e.g., Diffusion Policies, Transformers), these models often suffer from "hallucinations"—generating inaccurate actions or trajectories when faced with novel states not sufficiently covered by the training data. In closed-loop execution, such errors can be recursively amplified, leading to compounding errors that compromise safety and task success. Existing methods, such as scaling datasets or reinforcement learning fine-tuning, have not fully resolved the trade-off between generalization and reliability.

Methodology
The authors propose Curiosity-Diffuser, a method designed to guide conditional diffusion models to generate trajectories with lower "curiosity," thereby ensuring the generated behaviors remain within the distribution of the training data. The approach integrates three core components:

RND-Based Curiosity Evaluation: The method employs a Random Network Distillation (RND) module to assess the novelty of states. It consists of a fixed-parameter random encoder $f_\xi$ and a learnable prediction network $f_\eta$ . The "curiosity" value is defined as the prediction error $\|f_\eta(\tau) - f_\xi(\tau)\|^2$ . A lower error indicates the state is familiar (in-distribution), while a higher error suggests novelty (potentially OOD).
Curiosity-Guided Diffusion: Unlike methods that use RND merely to select from post-sampled trajectories, Curiosity-Diffuser incorporates the RND signal directly into the diffusion sampling process via classifier guidance. The guidance term $g$ combines a reward-based gradient ( $g_1$ ) and a curiosity-minimization gradient ( $g_2$ ):
$g = g_1 + \lambda g_2$
Here, $g_2$ is derived from the gradient of the RND prediction error, steering the denoising process away from high-curiosity (unreliable) regions while $g_1$ ensures task completion. The training of the policy network and the RND classifier is decoupled.
K-Sim Metric: To quantitatively evaluate reliability, the authors introduce K-Sim, a computationally efficient metric measuring the similarity between generated behaviors and the training dataset. It utilizes K-Means clustering on the training states to efficiently find the nearest state-action pairs for test trajectories, calculating a normalized distance score. A score closer to 1.0 indicates higher similarity to the training distribution.

Key Contributions

Curiosity-Guided Diffusion: A novel framework that uses RND to guide the conditional diffusion model during inference, actively suppressing OOD behaviors and reducing hallucinations rather than just filtering them post-hoc.
Reliability Evaluation Metric: The introduction of K-Sim, a scalable metric for measuring the closeness of policy decisions to the training dataset, facilitating research into stable imitation learning.
Decoupled Training: A strategy where the policy and the curiosity classifier are trained independently, allowing for flexible integration of guidance signals.
Comprehensive Validation: Extensive evaluations across simulation (D4RL MuJoCo, Minari AntMaze) and real-world robotic manipulation tasks.

Experimental Results

Simulation (MuJoCo): In the Medium-Expert dataset, Curiosity-Diffuser achieved the best average performance among baselines (77.9), showing particular strength in tasks with high-quality expert demonstrations. However, the authors note that performance gains are modest and data-dependent; in lower-quality datasets (Medium-Replay), the method showed sensitivity, sometimes underperforming compared to Conservative Q-Learning (CQL) or Implicit Q-Learning (IQL).
Simulation (AntMaze): The method demonstrated significant superiority in sparse-reward environments. While baselines like Diffuser and Decision Transformer failed to solve the AntMaze tasks (0% success in many configurations), Curiosity-Diffuser achieved success rates up to 58.6% (AntMaze-Medium-Play) and 42.0% (AntMaze-Large-Diverse). The RND guidance proved critical for navigating long-horizon tasks where reward signals are sparse.
K-Sim Analysis: Curiosity-Diffuser achieved the highest average K-Sim score (0.827) compared to Diffuser (0.785) and ReDiffuser (0.795), confirming that the guided generation process produces trajectories more similar to the training data.
Real-World Manipulation: In experiments with a PiPER robotic arm performing "Hammer Close Door" and "Block Ranking" tasks, Curiosity-Diffuser achieved an average success rate of 85%, outperforming standard Diffusion Policy (57.5%) and Action Chunking Transformer (47.5%). The method showed robustness against mild distribution shifts, such as randomized object placements and initial pose perturbations.
Sensitivity Analysis: The authors found a non-monotonic relationship between the curiosity weight $\lambda$ and task performance. Excessive reliance on curiosity guidance ( $\lambda$ too high) can hinder task completion, while $\lambda=0$ leads to hallucinations. An optimal balance is required.

Significance and Claims
The paper claims that Curiosity-Diffuser significantly improves the reliability of imitation learning policies by preventing the model from drifting into OOD regions during inference. By integrating curiosity minimization directly into the diffusion generation process, the method reduces compounding errors and hallucinations, leading to safer and more consistent robot behaviors. The authors emphasize that while the method improves performance, it is not a universal panacea; its effectiveness is contingent on data quality, and it requires careful tuning of the guidance weight. The work provides a practical pathway for deploying more robust imitation learning agents in real-world scenarios where safety and reliability are paramount.

The Story of the Curious Robot

Technical Summary: Curiosity-Diffuser

More like this