💻 computer science

Compositional Motion Generation from Demonstration with Object-Centric Neural Fields

This paper proposes a generative learning-from-demonstration framework that achieves scalable and data-efficient robotic motion generation by connecting perception and action through shared object-centric neural fields and a temporal mixture-of-experts mechanism, enabling systematic generalization across diverse scene configurations with minimal training data.

Original authors: Ahmet Ercan Tekden, Yasemin Bekiroglu

Published 2026-07-09

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ahmet Ercan Tekden, Yasemin Bekiroglu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you want to teach a robot how to tidy up a messy room. The old way of doing this is like showing the robot a single, long video of you cleaning and hoping it memorizes every single muscle movement. If you move a chair slightly differently next time, the robot gets confused and fails.

This paper proposes a smarter, more "compositional" way to teach robots. Instead of memorizing one giant, rigid script, the robot learns to break tasks down into smaller, reusable building blocks, much like a chef learning to chop, sauté, and plate separately before combining them into a full meal.

Here is how their system works, using simple analogies:

1. The "Object-Centric" Camera (Seeing the World)

Most robots look at a picture and see a giant, blurry soup of pixels. This paper teaches the robot to see the world as a collection of distinct, individual objects, like a child playing with LEGO bricks.

The Analogy: Imagine a transparent sheet of plastic with a drawing of a bowl on it, and another sheet with a tennis ball. You can slide the bowl sheet around, and the ball sheet around, independently.
How it works: The robot uses a special "neural field" (a type of AI brain) to separate the scene into these individual sheets. It learns a compact "code" (a latent vector) for each object that tells the robot where the object is and what it looks like. This allows the robot to understand that moving the bowl doesn't change the ball, making it very efficient at learning from just a few examples.

2. The "Conductor" (Mixing the Movements)

Once the robot sees the objects, it needs to decide how to move its arm. The authors use a system called a Mixture-of-Experts (MoE).

The Analogy: Think of a musical orchestra. You have different sections: strings, brass, and percussion. In a monolithic approach, the whole orchestra plays one giant, unchangeable song. In this new approach, a conductor (the gating mechanism) decides which section plays at which moment.
How it works:
- Expert 1 might know how to reach for a cup.
- Expert 2 might know how to pick up a ball.
- Expert 3 might know how to place an item in a bowl.
- As the task progresses (time passes), the "conductor" smoothly blends these experts together. If the robot needs to pick up a ball and then drop it, the conductor fades out the "reach" expert and fades in the "drop" expert. This happens automatically based on the objects present in the scene.

3. Learning from Few Examples (The "Sketch" vs. The "Photo")

Because the robot understands the structure of the scene (objects) and the structure of the movement (primitives), it doesn't need thousands of videos to learn.

The Analogy: If you teach a human to draw a cat by showing them one sketch, they can draw a cat in a different pose because they understand the concept of "cat-ness" (ears, tail, body). If you teach a robot by showing it 10,000 photos of cats, it might just memorize the pixels and fail if the cat is a different color.
The Result: The paper shows that their robot can learn complex tasks (like stacking cubes or picking up items) with as few as 10 to 30 demonstrations. Other methods that just look at raw images often need hundreds or thousands of tries to get the same result.

4. Real-World Testing (The "Magic" of Generalization)

The researchers tested this in a computer simulation and with a real robot arm.

The "Language" Trick: In one experiment, they didn't just show the robot a picture of a specific ball. They used a language tool to tell the robot, "Pick up the ball." The robot then successfully picked up a tennis ball during training and a baseball during testing, even though it had never seen a baseball before. It generalized the concept of "ball" rather than memorizing a specific texture.
The "3D" Trick: In another test, the robot had to open a drawer and put a box inside. The robot scanned the room in 3D (like a depth map) and figured out how to open the drawer and grab the box, even when the starting positions changed. It did this by interpolating (filling in the gaps) between the few examples it was shown.

Summary

In short, this paper introduces a robot learning system that:

Sees the world as separate, movable objects rather than a messy picture.
Moves by blending different "skills" (like reaching or dropping) together like a conductor mixing music.
Learns incredibly fast, needing very few examples to master new tasks.
Generalizes well, meaning it can handle new objects or slightly different room layouts without needing to be retrained from scratch.

The authors claim this makes robot learning much more efficient and robust, allowing robots to adapt to new situations with minimal human instruction.

Technical Summary: Compositional Motion Generation from Demonstration with Object-Centric Neural Fields

Problem Statement
Learning from Demonstration (LfD) faces a central challenge in representing diverse, long-horizon behaviors from a limited number of demonstrations. Treating motion as a monolithic sequence often overlooks the compositional structure of tasks, which typically unfold through sequential, object-level interactions (e.g., reaching, grasping, placing). While Movement Primitives (MPs) offer a compact representation of trajectories, their compositionality is often manually imposed, and they rely on hand-crafted, low-dimensional features that are difficult to estimate consistently. Conversely, learning features directly from images typically requires large datasets, sacrificing the data efficiency inherent to MPs. Furthermore, existing neural field-based approaches often operate at the scene level, scaling poorly with object count, or fail to support visual conditioning for full-trajectory generation.

Methodology
The authors propose a generative LfD framework that connects perception and motion through shared object-level representations. The approach consists of two primary stages:

Compositional Scene Modeling (Spatial MoE):
The framework models scenes using a spatial Mixture-of-Experts (MoE) based on object-centric neural fields. Each image is rendered as a composition of a fixed background and $N$ object-specific fields.
- Representation: Each object $i$ is represented by a latent vector $z_i$ and a coordinate-based MLP ( $F_i$ ) that predicts appearance. A deformation network ( $\Delta_i$ ), conditioned on $z_i$ , maps canonical coordinates to deformed coordinates to capture positional and geometric variations.
- Training: The model employs a soft mask-based importance sampling strategy. This ensures each expert learns independently without collapsing onto the same object, even if the input masks are coarse. Training proceeds in stages: background estimation, object-centric pretraining with a position-aware zoom curriculum, and joint compositional refinement.
- Latent Relabeling: To ensure consistency, the authors define a "corner set" $C_i$ for each object, representing the boundaries of observed variations. During training and inference, latent vectors are recovered by searching for convex combinations of these corners that best reconstruct the input image. This canonicalizes the latent space, constraining generalization to the demonstrated support.
Compositional Motion Generation (Temporal MoE):
Motion is generated using a temporal MoE that conditions trajectory generation on the object latent vectors.
- Gating Mechanism: A time-dependent gating mechanism computes weights $w_i(t)$ for each object expert. This allows different trajectory segments to be influenced by different subsets of objects (e.g., a "pick" phase depends on the object's location, while a "place" phase depends on the target).
- Conditioning: The framework utilizes Feature-wise Linear Modulation (FiLM) to inject object-specific information into the trajectory generator. The aggregated conditioning signal modulates the network parameters, enabling the model to learn smooth, object-conditioned movement primitives directly from visual input.
- Optimization: The model minimizes the negative log-likelihood of demonstrated trajectories, employing latent dropout during early training to prevent spurious dependencies.

Key Contributions

Object-Centric Generative Scene Model: A method to learn smooth, interpretable object-centric representations from images using a spatial MoE with latent-conditioned deformations.
Temporal Gating for Motion: A motion generation framework that employs a temporal MoE to achieve sequential compositionality, grounding motion in visual structure while maintaining the data efficiency of primitives.
Data-Efficient LfD Framework: An integrated system that learns movement primitives directly from visual input with significantly fewer demonstrations than image-based baselines, supporting generalization across diverse scene configurations.
3D and Real-World Validation: The framework is demonstrated to operate on 3D scene representations (via occupancy fields) and generalize at the category level using language-based segmentation models.

Experimental Results
The framework was evaluated in simulation and real-world robotic experiments:

Simulation: The method was tested on four manipulation tasks (Wall Avoidance, Incline Pick-and-Place, Cup Stacking, Cube Stacking) with varying object locations and geometric changes.
- Data Efficiency: The proposed method achieved low Mean Euclidean Distance (MED) and high success rates using 10–30 demonstrations. It consistently outperformed baselines (including Conditional Neural Movement Primitives, Diffusion Policy, and CNN-FiLM) trained with 3 to 10 times more data.
- Ablations: Removing temporal gating resulted in higher errors, confirming the value of time-varying latent aggregation. The object-centric approach outperformed global feature baselines (CNN-FiLM, DINO-FiLM).
Real-World Experiments:
- Robustness: The system successfully completed pick-and-place tasks with distractor objects and varying lighting.
- Category-Level Generalization: Using vision-language models for segmentation, the robot generalized to unseen objects within the same semantic category (e.g., different mugs or balls) without retraining.
- 3D Generalization: In a drawer-opening task, the model utilized 3D occupancy fields derived from ESDF scans. Trained on only four demonstrations, it successfully generalized to novel drawer opening angles and box positions, achieving sub-centimeter grasp accuracy.

Significance
The paper claims that compositional modeling at the object level provides a powerful paradigm for LfD. By coupling object-centric neural representations with a temporal mixture-of-experts, the framework offers interpretable latent structure and systematic generalization. The significance lies in its ability to achieve high task performance with limited demonstrations, bridging the gap between the data efficiency of traditional primitives and the perceptual richness of deep learning. The authors note that while the method is robust to moderate noise and distractors, it relies on the assumption that objects occupy distinct regions and that the scene layout remains consistent. Future work aims to extend the framework to incorporate neural radiance field-based scene representations for broader scene-level generalization.

1. The "Object-Centric" Camera (Seeing the World)

2. The "Conductor" (Mixing the Movements)

3. Learning from Few Examples (The "Sketch" vs. The "Photo")

4. Real-World Testing (The "Magic" of Generalization)

Summary

More like this