Fine-Tuning Robot Policies While Maintaining User Privacy

This paper introduces PRoP, a model-agnostic framework that enables robots to personalize their policies for individual users while preserving privacy by using unique cryptographic keys to mathematically transform network weights, ensuring that only authorized users can access their specific behavioral adaptations.

The Gist

Imagine you buy a high-tech, general-purpose robot chef. Out of the box, it knows how to make a "standard" burger. It's great, but maybe you prefer your burger with extra pickles, no onions, and the bun toasted just a little longer.

To make the robot yours, you teach it your specific preferences. You show it how you like your food. This is called fine-tuning.

The Problem: The "Open Kitchen" Privacy Leak
Here's the catch: In most current systems, once you teach the robot your secret recipe, that knowledge gets baked directly into the robot's brain. If someone else (like a neighbor, a hacker, or even the robot manufacturer) gets access to that robot, they can just turn it on, ask for a burger, and watch it make your specific version. They can instantly figure out that you hate onions and love extra pickles. Your private habits are leaked just by watching the robot work.

The Solution: PRoP (The "Magic Key" System)
The authors of this paper, Benjamin Christie, Sagar Parekh, and Dylan Losey, propose a new system called PRoP (Personalized and Private Robot Policies).

Think of PRoP like a smart lock with a master key and a unique personal key.

1. The Master Lock (The General Robot): The robot still has its original "general" brain. It knows how to make the standard burger. This is the default setting.
2. The Magic Key (Your Password): You have a unique key (it could be a password, a fingerprint, or even your face scan).
3. The Transformation: When you give the robot your key, the robot doesn't just "remember" your preferences. Instead, the key acts like a mathematical magic spell that temporarily reshuffles the robot's internal gears just for you.
   • With your key: The gears shift, and the robot makes your perfect burger with extra pickles.
   • Without your key (or with the wrong one): The gears snap back to their original position. The robot makes the standard burger. It has no idea you exist or what you like.

How It Works (The "Secret Sauce" Analogy)
Usually, to teach a robot a new trick, you have to rewrite its entire instruction manual (the neural network weights). This is messy and makes the robot forget its old tricks or leak your data.

PRoP is clever because it doesn't rewrite the manual. Instead, it uses your key to tweak the intermediate steps of the robot's thinking process.

• Imagine the robot's brain is a long assembly line making a sandwich.
• PRoP puts a special, invisible filter on the conveyor belt.
• When your key is present, the filter rearranges the ingredients on the belt to match your taste.
• When the key is gone, the filter disappears, and the ingredients go back to the standard arrangement.

Because the robot's core "brain" (the assembly line) never actually changes permanently, it stays safe. Even if a hacker steals the robot's code, they can't see your preferences because your preferences are hidden inside the interaction between your key and the robot's gears, not stored in the gears themselves.

Why This is a Big Deal
The researchers tested this on robots doing everything from making sandwiches to sorting images. They found that:

• It's Private: If you try to guess someone else's key (even if you get it 99% right), the robot won't reveal their secrets. It just goes back to being a "normal" robot.
• It's Efficient: You don't need a separate robot brain for every person. One robot can serve 100 different people, each with their own secret preferences, without getting confused or leaking data.
• It's Flexible: It works whether the robot is learning by watching you (Imitation Learning), learning by trial and error (Reinforcement Learning), or just sorting pictures.

In a Nutshell
PRoP is like giving your robot a personalized "mode" button that only you can activate. It lets the robot be "you" when you are there, but keeps it a "stranger" to everyone else. This way, you can have a robot that truly understands your habits without ever having to worry that your secrets will be spilled to the world.

Technical Summary

Here is a detailed technical summary of the paper "Fine-Tuning Robot Policies While Maintaining User Privacy" by Christie, Parekh, and Losey.

1. Problem Statement

The rise of general-purpose robot policies (pre-trained models capable of performing various tasks) necessitates fine-tuning to adapt to individual user preferences (e.g., specific food preparation styles or task objectives). However, current fine-tuning methods create a significant privacy vulnerability:

• The Leakage Issue: When a robot's policy is fine-tuned for a specific user, the resulting model encodes that user's private preferences. Any agent with access to this fine-tuned model can "roll out" the policy (execute it) and infer the original user's habits, tastes, or behaviors.
• Limitations of Existing Solutions:
  • Data Privacy: Protecting the training dataset does not prevent inference from the final model.
  • Model Privacy (Encryption): Techniques like homomorphic encryption are computationally infeasible for real-time robotics (up to $10^7$ times slower).
  • Differential Privacy: Often struggles with the "privacy budget" ($\epsilon$) in complex interaction tasks and does not gate access to specific behaviors.
• Core Challenge: How to enable a robot to learn and adapt to individual users while ensuring that unauthorized users (or third parties) cannot access or infer those specific preferences, even if they possess the trained model.

2. Methodology: PRoP (Personalized and Private Robot Policies)

The authors propose PRoP, a model-agnostic framework that achieves private personalization without altering the original network architecture.

Core Concept: Key-Based Latent Transformation

Instead of training separate models for each user or adding the user key as a standard input (which requires architectural changes), PRoP uses a unique user key ($k$) to mathematically transform the intermediate features of the pre-trained policy.

• The Key ($k$): A unique identifier for the user (e.g., a password, biometric feature, or token), represented as a bit-vector.
• Key Encoder ($\Delta_\phi$): A Multi-Layer Perceptron (MLP) that maps the user key $k$ into a latent space $Z$.
• Affine Transformation: The latent encoding is used to perform an affine transformation on the weights of specific hidden layers in the robot's policy network ($R_\phi$).
  • For a hidden layer $i$ with weights $W_i$ and bias $b_i$, and latent vector $\delta_i = \Delta_\phi(k)$, the output $z_{i+1}$ is calculated as:
    $$z_{i+1} = f(W_i \cdot \text{diag}(\delta_i) \cdot z_i + b_i)$$
  • This effectively "gates" the policy. Without the correct key, the transformation defaults to the identity (or a neutral state), reverting the robot to its general baseline behavior ($\pi^*$). With the correct key, the policy shifts to the personalized behavior ($\pi'$).

Training Objective

The training process optimizes two competing goals simultaneously using a composite loss function:

1. Personalization: Minimize loss for the specific user's objective ($J'$) when the correct key $k'$ is provided.
2. Privacy/Fallback: Maximize performance on the general objective ($J^*$) for all other keys (including random keys and keys close to the correct one).

To make training tractable, the authors use an inductive loss function that samples:

• The correct key ($k'$).
• A subset of "close" keys (Hamming distance $\le \epsilon$) to enforce high-margin separation.
• A stochastic subset of random keys to ensure the model reverts to the general policy for unauthorized users.

3. Key Contributions

1. Key-Based Personalization Formulation: A novel method to condition robot policies on user keys without modifying the pre-trained architecture. This avoids the need to retrain the entire network or change input dimensions.
2. Privacy-Preserving Mechanism: The method ensures that unauthorized users (or those with incorrect keys) only see the general policy. The personalized behavior is mathematically "obfuscated" within the network weights, making it inaccessible without the specific key.
3. Model Agnosticism: PRoP is compatible with various learning paradigms, including Imitation Learning, Reinforcement Learning (RL), and Image Classification. It works with pre-trained models and can also be trained end-to-end.
4. Scalability: The approach allows multiple user preferences to be "compressed" into a single shared network, avoiding the memory overhead of storing separate models for every user.

4. Experimental Results

The authors evaluated PRoP against baselines (standard MLP with key input, Conditional VAE) across four domains:

• Imitation Learning (3-DoF Arm): PRoP successfully learned user-specific trajectories. It maintained high performance for the correct key while reverting to the general policy for incorrect keys.
• Reinforcement Learning (PandaGym): In a dense reward environment, PRoP achieved substantial performance gaps over baselines for correct keys while maintaining general task performance for incorrect keys.
• Image Classification (MNIST): The model learned to shift label predictions based on the key (e.g., $l \to (l+k) \mod 10$). PRoP showed significantly lower information leakage for "close" keys (1-bit difference) compared to baselines.
• Scalability (Multiple Users): PRoP could handle up to ~16 unique users with a single network before performance began to decay linearly. In contrast, baselines (MLP/CVAE) showed exponential performance decay as the number of users increased.
• Real-World User Study: In a mock kitchen setting with 12 participants and a UR-10 robot:
  • Personalization: PRoP outperformed baselines in correctly assembling sandwiches according to the user's password.
  • Privacy: PRoP exhibited significantly lower privacy leakage (users could not infer other users' orders) compared to CVAE and MLP ($p < 0.05$).

5. Significance and Conclusion

This work addresses a critical gap in Human-Robot Interaction (HRI): the tension between personalization and privacy.

• Practical Impact: PRoP enables the deployment of general-purpose robots in domestic settings where users can customize behavior without fear of their private habits being exposed to manufacturers, third parties, or other users.
• Technical Innovation: By leveraging intermediate feature transformation rather than input concatenation or full model duplication, PRoP offers a computationally efficient, real-time solution that preserves the integrity of pre-trained models.
• Future Direction: The paper suggests that this approach could be extended to language-based personalization and more complex obfuscation techniques, paving the way for safe, scalable, and personalized robotic assistants.