💻 computer science

FOCI Policy: Focus on Object-Centric Interactions for Relational Manipulation Policies

The paper proposes the FOCI Policy, an object-centric framework that improves generalization and data efficiency in rigid relational manipulation by abstracting skills into temporally compact interaction segments and spatially invariant relative $SE(3)$ motions between task-relevant objects.

Original authors: Ze Fu, Pinhao Song, Yutong Hu, Renaud Detry

Published 2026-09-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ze Fu, Pinhao Song, Yutong Hu, Renaud Detry

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Robots have long struggled with the simple act of moving objects from one place to another. While they can be programmed to perform repetitive tasks in a factory, teaching them to handle new situations with a single glance or a brief demonstration remains a profound challenge. Most current approaches try to teach a robot by showing it exactly how to move its own arms and fingers, essentially drilling a form of muscle memory. This method requires vast amounts of data, often hundreds of demonstrations, to cover every possible way an object might be positioned or a room might be arranged. If the robot encounters a slight change, such as a cup being moved a few inches to the left, the rigid instructions often fail. Researchers are now exploring a different path: instead of focusing on the robot's body, they are teaching machines to focus on the relationship between the objects themselves. By understanding how a toothbrush moves relative to a cup, rather than how the robot's gripper moves through space, a system might learn to generalize skills with far less effort.

A team of researchers at KU Leuven has developed a new framework called FOCI Policy that puts this idea into practice. Their work suggests that many complex tasks are governed by short, critical moments where objects interact, while the rest of the movement is merely travel. Imagine trying to learn how to pour a glass of water. The act of lifting the pitcher and walking to the table can be done in countless ways, but the moment the water flows from the spout into the glass is tightly constrained and follows a specific pattern. The researchers observed that if a robot can isolate these brief, high-stakes interaction phases and ignore the noisy, variable travel time, it can learn the task much more efficiently. They built a system that automatically scans a demonstration video, identifies exactly when the objects begin to touch or influence one another, and extracts that specific segment.

Once this critical segment is isolated, the system does not try to memorize the robot's exact path. Instead, it learns the relative motion between the two objects involved. If a human demonstrates placing a wine bottle into a rack, the robot learns the movement of the bottle relative to the rack, not the movement of the robot's arm. This approach makes the skill independent of the robot's specific body shape or the exact layout of the room. The system can then take this learned relationship and apply it to a new situation, even if the objects are in different positions or the robot is a different model entirely. In their tests, the researchers trained the system on just a single demonstration for each task. When faced with new scenarios, the robot successfully performed complex actions like stacking wine bottles, screwing in nails, or pouring liquids, often outperforming other methods that were trained on significantly more data.

The results were particularly striking when the visual conditions changed. In one set of experiments, the researchers introduced severe visual disturbances, such as changing the lighting, adding distracting objects, or altering the texture of the items. While other advanced systems trained on a hundred demonstrations saw their success rates drop dramatically under these conditions, the new method maintained a level of performance that was comparable to those heavily trained models, despite using only one-tenth of the data. This suggests that by focusing on the geometric relationship between objects, the robot becomes less confused by visual noise. The system proved robust enough to handle real-world tasks on a physical robot arm, successfully completing tasks like sweeping dust or opening a drawer after seeing the action just once.

However, the researchers are careful to note that this approach is not a universal solution for every type of movement. The method works best for tasks involving rigid objects that have clear, distinct phases of interaction, such as inserting a peg into a hole or placing a book on a shelf. It is less effective for tasks that involve continuous contact, like pushing a soft object across a table, or for situations where the objects are so small or hidden that the robot cannot reliably determine their position. The system relies on the ability to see the objects clearly; if the robot cannot estimate where an object is, it cannot calculate the correct relative motion. Despite these limitations, the findings offer a compelling shift in how robots learn. By stripping away the unnecessary details of how a robot moves and focusing on the essential dance between objects, the researchers have shown that machines can acquire new skills with a fraction of the data previously thought necessary. This efficiency brings us closer to a future where robots can learn new tasks in minutes rather than days, adapting quickly to the unpredictable nature of the real world.

Technical Summary: FOCI POLICY

Problem Statement

Robotic manipulation policies typically learn a direct mapping from observations to robot actions (action-centric). While effective in controlled settings, these approaches often require large datasets to cover the high-dimensional robot state space and struggle to generalize to novel object configurations or scene layouts. Existing object-centric methods attempt to improve generalization by modeling intermediate object representations (e.g., static goal poses or dense motion flows). However, static goal poses often fail to capture essential interaction dynamics, while dense motion flows are computationally expensive to learn and prone to drift.

The core challenge addressed by this paper is relational manipulation: tasks where success depends on the geometric and functional relationships between objects (e.g., inserting a toothbrush into a cup). The authors observe that many such tasks are governed by short, tightly constrained interaction phases, while the preceding transport phases are relatively unconstrained and task-irrelevant. Current methods often fail to isolate these critical phases, leading to inefficient learning and poor generalization, particularly in few-shot (single-demonstration) scenarios.

Methodology: FOCI POLICY

The authors propose FOCI POLICY (Focus on Object-Centric Interactions), an interaction-centric framework that abstracts manipulation skills into two dimensions: temporal and spatial.

1. Temporal Abstraction: Interaction Segments

Instead of learning from entire demonstration episodes $[0, H]$ , the policy automatically extracts interaction segments $\tau_{rel}$ within a sub-interval $[t_s, t_g]$ .

Detection Mechanism: The system treats segment identification as a Change-Point Detection (CPD) problem. It constructs a multivariate time series $\tilde{S}(t)$ combining gripper linear velocity, angular velocity, and the distance between the source and target objects.
Algorithm: Using the PELT algorithm with an RBF kernel cost function, the system identifies the optimal transition point $t_s$ where unconstrained transport ends and constrained interaction begins. This isolates the functionally critical phase without manual annotation.

2. Spatial Abstraction: Object-Centric Formulation

The policy represents the extracted interaction segment not as absolute robot trajectories, but as relative SE(3) motion between task-relevant objects.

Definition: Let $T_{src}$ and $T_{tgt}$ be the world-frame poses of the source object (e.g., the grasped tool) and the target object (e.g., the container). The interaction segment is defined as the relative motion:
$\tau_{rel} = \{ (T_{tgt}^{(k)})^{-1} T_{src}^{(k)} \}_{k=1}^n$
Invariance: This formulation renders the learned skill invariant to scene configurations and robot embodiment. The policy learns the relative motion required to solve the task, regardless of where the objects are initially placed or which robot is executing the task.

3. Policy Architecture

The model predicts the interaction segment conditioned on object point clouds ( $P_{src}, P_{tgt}$ ) and language instructions ( $\ell$ ).

Encoder: Point clouds are encoded using PointNet++. Features are fused via cross-attention to capture interaction-relevant geometry. Language embeddings (from CLIP) are concatenated with geometric features.
Two-Stage Prediction:
1. Goal Prediction: A Gaussian Mixture Model (GMM) predicts the terminal relative pose $T_{rel}^{(n)}$ , allowing for multi-modal interaction strategies (e.g., different insertion angles).
2. Trajectory Generation: A trajectory decoder generates intermediate waypoints conditioned on the predicted goal.
Action Recovery: At inference, the predicted relative trajectory is transformed back to the world frame using the estimated target object pose and a fixed offset ( $T_{ofs}$ ) to generate executable end-effector commands.

Key Contributions

Interaction as a Compact Abstraction: The paper demonstrates that task-relevant Object-Object Interactions (OOIs) provide a compact abstraction for relational manipulation. They propose an automatic pipeline using change-point detection to extract these segments from demonstrations, removing task-irrelevant variability.
Interaction-Centric Policy Architecture: A two-stage policy is designed that models manipulation as relative SE(3) motion between objects. This yields invariance to robot embodiment and scene configurations, addressing the limitations of both static goal poses and dense flow fields.
Data-Efficient Generalization: The resulting method achieves strong performance with a single demonstration and a single camera. It outperforms prior object-centric and action-centric baselines in simulation and real-world settings, suggesting that effective relational manipulation does not require large-scale training data.

Experimental Results

The authors evaluated FOCI POLICY on three benchmarks: RLBench, COLOSSEUM, and real-world tasks.

RLBench (Relational Manipulation): On 8 tasks with clear OOIs, FOCI POLICY trained on a single demonstration achieved significantly higher success rates than baselines (e.g., Imagination Policy, 3D Diffuser Actor, RVT2) trained on 5 demonstrations. Notably, on precision-critical tasks like Put-Roll and Put-Umbrella, FOCI POLICY achieved ~45% and ~38% success respectively with 1 demo, while action-centric baselines often failed (0-9%).
COLOSSEUM (Visual Robustness): Under severe visual perturbations (texture, lighting, distractors, camera pose), FOCI POLICY demonstrated robustness comparable to Vision-Language-Action (VLA) models (BridgeVLA, ActiveVLA) that were trained with 100 demonstrations per task. Despite using only 1% of the data, FOCI POLICY outperformed these VLA models in the "All Perturbations" setting (25.3% vs. ~20%).
RLBench-18 (Generalization): In a one-demonstration setting across 18 tasks, FOCI POLICY achieved a 43.7% average success rate. Performance was high (79.6%) on "easy-to-ground" tasks but dropped on "hard-to-ground" tasks (7.8%), indicating that the bottleneck is reliable interaction grounding (pose estimation) rather than the policy formulation itself.
Real-World Deployment: On a Franka Emika Panda with a single RGB-D camera, the policy achieved >53% task success across five diverse tasks (e.g., Insert-Tube, Pour-Liquid) using only one demonstration. It successfully transferred to a different robot embodiment (Robotiq gripper) and handled obstacle avoidance via classical motion planning integration.

Significance and Claims

The paper argues that for rigid relational manipulation, modeling object-object interactions provides a simple and efficient inductive bias. By focusing on the "interaction segment"—the specific interval where relative motion is constrained—the method achieves:

Data Efficiency: Learning complex skills from a single demonstration.
Generalization: Robustness to novel object poses, scene configurations, and even different robot embodiments.
Simplicity: The framework decouples the learning of interaction dynamics from the complexities of absolute robot control and full-trajectory imitation.

The authors conclude that effective relational manipulation does not necessarily require large-scale training data if the policy is explicitly designed to model the constrained relative motion between objects. They acknowledge limitations in scenarios involving continuous contact (e.g., pushing) or tasks with ambiguous geometry/occlusion where pose estimation fails, but posit that the interaction-centric formulation is a powerful step toward data-efficient robotic learning.