PRISM: Personalized Refinement of Imitation Skills for Manipulation via Human Instructions

PRISM is an instruction-conditioned framework that integrates imitation learning with reinforcement learning and human feedback to efficiently refine generic robotic manipulation policies into robust, fine-grained behaviors for new goals and constraints.

The Gist

Imagine you are teaching a robot to do a chore, like picking up a cup and putting it on a shelf.

The Problem with Old Methods:
Traditionally, you have two bad options:

1. The "Copycat" (Imitation Learning): You show the robot exactly how to do it once. It learns quickly, but it's like a parrot. If you move the cup slightly or ask it to hold the cup differently, the robot panics and drops it because it only knows the exact path you showed it.
2. The "Trial-and-Error" (Reinforcement Learning): You tell the robot, "Figure it out!" and let it crash into things millions of times until it learns. This makes the robot very smart and adaptable, but it takes forever and is dangerous (imagine a robot smashing your kitchen while learning).

The PRISM Solution:
The paper introduces PRISM, a new way to train robots that combines the best of both worlds. Think of PRISM as a smart apprenticeship program where a human mentor guides a talented but inexperienced apprentice.

Here is how it works, step-by-step, using a cooking analogy:

1. The "Base Recipe" (Imitation Learning)

First, a non-expert human (like a home cook) shows the robot how to do a basic task, like "Pick up a pot and toss it into the cupboard."

• The Analogy: The robot watches the human cook and learns a "base recipe." It gets good at the general motion but isn't perfect yet. It's like a junior chef who knows how to chop onions but might burn the sauce if the heat changes.

2. The "Smart Critic" (The LLM & Eureka)

Now, the human wants to change the task. Instead of tossing the pot, they want the robot to place a hot pot on a table without spilling the soup inside.

• The Analogy: The human tells the robot, "Hey, don't toss it! Keep it upright!"
• In the past, a programmer would have to write complex math code to explain why keeping it upright is good. With PRISM, the human just speaks naturally.
• The system uses a "Smart Critic" (an AI language model) that translates your English sentence ("Keep it upright") into a set of rules (a reward function) the robot understands. It's like a translator turning your complaint into a checklist for the chef.

3. The "Taste Test" (Human Feedback Loop)

This is the secret sauce. The robot tries the new task. Sometimes it fails (it spills the soup).

• The Analogy: The human tastes the soup and says, "Too salty!" or "It's burning!"
• In PRISM, the human gives sparse feedback (just a few comments) on specific moments where the robot messed up. The "Smart Critic" uses these comments to instantly update the rules.
• Instead of the robot crashing 1,000 times to learn, it learns from just a few corrections, guided by the human's voice.

4. The Result: A Personalized Master Chef

By the end, the robot has:

• The muscle memory from the first demo (it knows how to move).
• The adaptability from the trial-and-error phase (it knows how to recover if it slips).
• The personalization from your instructions (it knows your specific way of holding the pot).

Why is this a big deal?

• It's Fast: It doesn't need millions of tries. It learns like a human apprentice who learns from a few corrections.
• It's Safe: It starts with a safe, basic behavior and only tweaks it, so it doesn't go crazy and break things.
• It's for Everyone: You don't need to be a robot engineer or a mathematician. You just need to speak English and give a few pointers.

In a nutshell: PRISM is like hiring a robot that already knows the basics of cooking, then letting you (the non-expert) give it a few spoken instructions to customize the dish to your exact taste, without having to retrain the robot from scratch.

Technical Summary

Here is a detailed technical summary of the paper "PRISM: Personalized Refinement of Imitation Skills for Manipulation via Human Instructions."

1. Problem Statement

Robotic manipulation in unstructured environments faces a fundamental trade-off between data efficiency and robustness/adaptability:

• Imitation Learning (IL): While data-efficient and capable of rapid initialization from human demonstrations, IL policies are brittle. They struggle with out-of-distribution (OOD) events, lack recovery strategies, and fail to generalize to new goal configurations or constraints (e.g., changing a "toss" task to a "place" task).
• Reinforcement Learning (RL): RL offers robustness and the ability to discover reactive behaviors through exploration. However, training RL from scratch is sample-inefficient, requires complex reward engineering, and is often impractical for real-world deployment due to safety and time constraints.
• The Gap: Existing hybrid methods often rely on manually engineered rewards or assume known task shifts. There is a lack of frameworks that allow non-expert users to easily personalize policies for new goals and constraints using natural language, without requiring continuous supervision or expert reward design.

2. Methodology: The PRISM Pipeline

PRISM (Personalized Refinement of Imitation Skills for Manipulation via Human Instructions) is a modular pipeline that bridges IL and RL, guided by natural language and sparse human feedback. It consists of three main stages:

A. Data Collection & Imitation Learning (Initialization)

• Data Source: Teleoperated demonstrations are collected from non-expert users (using VR setups) to perform a generic task (e.g., pick and toss).
• Model: The demonstrations are distilled into an initial policy using Behavioral Cloning (BC) with a Recurrent Gaussian Mixture Model (BC-GMM-RNN) via the Robomimic framework.
• Output: A "Generic Policy" ($\pi_{BC}$) that serves as a behavioral prior. This policy is competent in the demonstrated distribution but brittle to shifts.

B. Reinforcement Learning Refinement (Task Adaptation)

The generic policy is refined using Proximal Policy Optimization (PPO) to adapt to new task specifications (e.g., changing target poses or adding constraints).

• Behavior-Matching Regularization: To prevent the policy from drifting too far from the safe, demonstrated behaviors, a regularization term is added to the PPO loss function. This encourages the refined policy to stay close to $\pi_{BC}$ when observing similar states, preserving the "safety" of the initial imitation.
• Reward Generation (Eureka Integration): Instead of manual reward engineering, PRISM utilizes the Eureka framework. A Large Language Model (LLM) parses natural language instructions (e.g., "keep the cup upright") and automatically generates structured reward functions.

C. Personalization via Human-in-the-Loop

To handle complex personalization and ensure alignment with user intent, PRISM employs a hybrid prompting loop:

1. Automated Iteration: The LLM iteratively generates and refines reward candidates based on automated evaluation metrics (success rates, constraint violations).
2. Sparse Human Feedback: At predefined checkpoints, a non-expert user provides natural language corrective feedback on intermediate rollouts (e.g., "In rollout A, the object was placed correctly but wasn't kept vertical").
3. Reward Update: The LLM incorporates this feedback to update the reward function, steering the RL agent toward the specific user preference.

3. Key Contributions

1. Instruction-Conditioned Refinement: A novel pipeline that allows non-expert users to modify and personalize robotic policies using natural language instructions rather than code or complex reward engineering.
2. Hybrid Human-AI Feedback Loop: A mechanism that combines automated LLM-based reward generation with sparse, targeted human corrective prompts. This reduces the need for continuous supervision while ensuring the policy aligns with specific user constraints.
3. Behavior-Matching Regularization: A technique that integrates IL priors into the RL objective, ensuring sample efficiency and preventing the agent from learning "reward-hacking" behaviors that violate the original safety priors.
4. End-to-End Pipeline: A seamless integration of teleoperation, BC, and instruction-guided RL, demonstrated on a simulated manipulation task.

4. Experimental Results

The method was evaluated in a simulated environment (IsaacSim/Omniverse) using a pick-and-place task with specific constraints.

• Baseline Comparison:
  • IL Only: Achieved only 21.2% success rate on the generic task and failed to generalize to new constraints.
  • RL Only (Eureka without IL): Failed to learn the task even after 15,000 steps (10 iterations), highlighting the necessity of the IL initialization.
  • PRISM (IL + RL + Feedback): Achieved a 96.8% success rate.
• Efficiency:
  • The full personalization process (adapting from a "toss" to a "vertical place" task) took approximately 4 hours total.
  • PRISM significantly outperformed fully automated RL approaches in convergence speed and final performance.
• Impact of Human Feedback:
  • Experiments showed that sparse human feedback (introduced every 5 automated iterations) drastically accelerated convergence and reduced variability compared to fully automated reward generation.
  • The method successfully adapted the policy to maintain an object's verticality during transport, a constraint the initial IL policy could not satisfy.

5. Significance and Future Work

Significance:
PRISM demonstrates that combining the data efficiency of Imitation Learning with the adaptability of Reinforcement Learning, guided by natural language and sparse human feedback, creates a viable path for deployable, user-adaptive robotic systems. It lowers the barrier for non-experts to customize robot behaviors, addressing the "brittleness" of pure IL and the "sample inefficiency" of pure RL.

Limitations & Future Directions:

• Simulation Only: Current results are limited to simulation; real-world dynamics, perception noise, and hardware constraints remain unvalidated.
• Scalability: The current protocol relies on explicit success criteria and occasional human feedback, which may need refinement for long-horizon tasks or diverse user bases.
• Future Work: The authors plan to close the sim-to-real gap, evaluate scalability across varied task families, and develop implicit preference inference mechanisms to reduce reliance on explicit user feedback.