💻 computer science

Joint Discovery of Object and Action Symbols through Effect Prediction for Robotic Manipulation Planning

This paper proposes a framework that jointly discovers discrete object and action symbols by predicting multi-modal interaction effects from random data, enabling robust few-shot generalization and precise manipulation planning for both seen and novel objects based on behavioral rather than visual similarity.

Original authors: Burcu Kilic, Berke Kartal, Fatih Dogangun, Erhan Oztop, Emre Ugur

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Burcu Kilic, Berke Kartal, Fatih Dogangun, Erhan Oztop, Emre Ugur

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine teaching a robot to move objects around a table. If you just show it pictures of blocks, balls, and cylinders, the robot might think a round ball and a round donut are the same because they look alike. But if you try to push them, the ball rolls away while the donut stays put. The robot needs to learn that what an object does is more important than what it looks like.

This paper presents a new way for robots to learn this lesson on their own, without a human teacher telling them what every object is. Here is how it works, broken down into simple concepts:

1. The "Blindfolded" Learning Game

Instead of just looking at objects, the robot plays a game of "blindfolded exploration." It randomly grabs, pushes, lifts, and drops various objects while recording everything that happens:

Where the object moved.
How hard the robot had to push or pull (force).
Whether the object slipped or stuck (contact).

Think of this like a baby learning about the world. A baby doesn't just look at a rattle; they shake it, drop it, and bite it to understand how it behaves. This robot does the same thing, but with sensors.

2. The "Secret Code" (Symbols)

The robot takes all this messy data and compresses it into a simple "secret code" (binary symbols).

Object Code: It groups objects not by shape, but by how they react. For example, it learns that a "Cube" and a "T-Block" might get the same code if they both require a specific grip to lift, even if they look different.
Action Code: It groups its movements. It learns the difference between a "push" and a "pick-up," and even the difference between a "push to the left" and a "push to the right."

The robot uses a special two-step training process to learn this:

Step 1 (The Rough Draft): It first learns to tell the difference between big actions (like "pushing" vs. "lifting") just by feeling the force.
Step 2 (The Fine Print): Then, it refines this to learn the specific directions and details, like exactly how far the object moves.

3. The "Map and Compass" (Planning)

Once the robot has these codes, it builds a discrete map (a library of effects).

Imagine a chessboard where every square is a possible position for an object.
The robot knows: "If I use Action Code A on Object Code B, the object will move to Square X."
It uses a search algorithm (like a GPS finding the shortest route) to string these moves together to get from Point A to Point B.

Crucially, the robot doesn't just plan the final destination; it plans the middle steps. It can say, "I'll push it halfway, check where it is, and then adjust." This allows for precise control, unlike other methods that just guess the whole path at once.

4. The "Few-Shot" Superpower

The most impressive part is how the robot handles new objects it has never seen before.

Old Way: If you show a robot a new "T-Block," it looks at the picture. If it looks like a "Cube," it tries to treat it like a cube. If the T-Block is heavy or slippery, the robot fails.
This Paper's Way: The robot tries to push or lift the new T-Block just three times. It compares the results (the force and movement) to its existing "secret codes."
- It realizes, "Hey, this new T-Block reacts exactly like the 'Block X' I already know!"
- It assigns the T-Block the same code as Block X and uses its existing map to move it perfectly.

The Results

The researchers tested this in a simulation with tasks like moving objects to a specific spot and stacking them.

Better Accuracy: Their method was much more precise at moving objects to the right spot compared to a popular "Diffusion Policy" (a type of AI that learns by guessing and correcting).
Better Stacking: When stacking blocks, the robot succeeded far more often because it understood how to grip different shapes, whereas the other method often dropped or knocked over the blocks.
New Objects: When given brand new shapes (like a donut or a U-shape), the robot learned to handle them correctly after just a few tries, while the visual-based methods failed because they were fooled by the shapes.

In a Nutshell

This paper teaches robots to stop being "photographers" (who only care about how things look) and start being "tactile explorers" (who care about how things feel and move). By learning the "physics" of objects through trial and error, the robot can plan complex moves and instantly adapt to new toys it has never seen before.

Technical Summary: Joint Discovery of Object and Action Symbols through Effect Prediction for Robotic Manipulation Planning

Problem Statement
Autonomous robots face a fundamental challenge in long-horizon manipulation tasks: reasoning over high-dimensional, continuous sensorimotor spaces is computationally unmanageable. Effective planning requires abstracting continuous percepts and actions into discrete representations. Previous approaches have limitations: visual-based categorization fails to distinguish objects that look similar but behave differently, while interaction-based methods often rely on predefined actions or ignore the temporal structure of trajectories. Furthermore, existing few-shot generalization methods typically rely on static geometric or visual similarity, failing to capture the physical dynamics of interactions (affordances) that are critical for handling novel objects. The core problem addressed is how to autonomously discover high-level object and action symbols from continuous, random interaction data to enable precise, multi-step planning and robust generalization to unseen objects.

Methodology
The authors propose a neuro-symbolic framework that jointly discovers object and action symbols through a two-stage learning process optimized on multi-modal effect prediction.

Architecture and Symbol Discovery:
- The system employs an encoder-decoder network with a binary bottleneck layer.
- Object Encoder ( $\phi_o$ ): A convolutional neural network maps object depth images to binary symbols ( $z_o$ ).
- Action Encoder ( $\phi_a$ ): A Transformer architecture maps joint angle trajectories to binary symbols ( $z_a$ ), capturing temporal structure.
- Binary Activation: Both encoders use a Gumbel-Sigmoid (GS) activation function to produce discrete binary vectors, allowing gradients to pass during training via the reparameterization trick.
- Decoder ( $\psi$ ): Predicts the multi-modal effect trajectory (object position, force vectors, and contact feedback) given the concatenated object and action symbols.
Two-Stage Training Procedure:
- Stage 1: The model predicts only force and contact feedback effects using full object bits and a subset of action bits. This stage learns to distinguish coarse high-level primitives (e.g., push vs. pick-and-place) based on tactile feedback, mimicking early infant sensorimotor development.
- Stage 2: The model uses all symbol bits to predict the entire effect trajectory, including spatial position changes. This stage refines action symbols to capture directional variations (e.g., left push vs. right push).
Planning Framework:
- A discrete effect library is constructed from the learned model, mapping tuples of (object symbol, action symbol, timestep) to predicted position effects.
- A Search:* Planning is performed via A* search over this library. The algorithm utilizes partial action executions, where intermediate timesteps in the predicted trajectory serve as distinct reachable states, allowing for precise low-level control.
- Execution: Symbolic plans are converted into executable joint-angle trajectories using a Symbol-Conditioned Low-Level Controller (Conditional Neural Movement Primitives, CNMP).
- Drift Validation: A replanning mechanism is triggered if the actual object position deviates from the predicted position beyond a threshold, ensuring execution integrity.
Few-Shot Generalization:
- Novel objects are categorized not by visual appearance but by behavioral similarity.
- The system collects a small number of random demonstrations for a new object, extracts action symbols, and predicts effects for candidate object symbols.
- The object symbol with the lowest mean prediction error across demonstrations is assigned to the novel object, enabling planning without retraining.

Key Contributions

Joint Symbol Discovery: A deep encoder-decoder network with a binary bottleneck that jointly discovers object and action symbols through a two-stage learning procedure optimized on multi-modal (spatial-haptic) effect trajectory predictions.
Discrete Planning with Partial Execution: A planning framework that builds a discrete effect library and performs A* search using intermediate trajectory points for precise control, combined with symbol-conditioned low-level controllers and drift-based replanning.
Effect-Driven Few-Shot Generalization: A method for categorizing novel objects based on behavioral similarity (effect matching) rather than visual similarity, enabling manipulation planning on unseen objects with minimal demonstrations.

Experimental Results
Experiments were conducted in a PyBullet simulation environment using a UR10e manipulator and five object types (Ball, Cube, Block X, Y, Z). The framework was evaluated against a Diffusion Policy baseline on two tasks: tabletop repositioning (Task 1) and stacking (Task 2).

Effect Prediction: The proposed two-stage Transformer-based architecture achieved significantly lower position prediction errors (1.70 cm) compared to single-stage learning (1.77 cm) and fully connected baselines (1.82 cm).
Planning Performance (Seen Objects): The proposed method significantly outperformed the Diffusion Policy baseline in both tasks. In Task 1, it achieved lower Euclidean distance errors across all object types (e.g., 4.49 cm vs. 5.09 cm for the Cube). In Task 2 (stacking), it achieved significantly higher success rates (e.g., 69.67% vs. 18.33% for Block Z), attributed to its ability to learn grasp affordances and intermediate states.
Few-Shot Generalization (Novel Objects): When tested on five novel objects (Torus, T-Block, Cylinder, U-Block, Wide Block) with only three demonstrations each, the effect-driven approach significantly outperformed the Diffusion Policy and a visual-based encoding baseline. Notably, the visual baseline misclassified objects (e.g., categorizing a Torus as a Ball), whereas the effect-based method correctly identified behavioral similarities (e.g., Torus behaving like a Cube in terms of push effects), leading to superior planning success.

Significance and Claims
The paper claims that autonomous agents must discover representations through interaction to handle novel scenarios effectively. The significance of this work lies in demonstrating that:

Interaction over Appearance: Categorizing objects and actions based on multi-modal interaction effects (force, contact, motion) is superior to visual-based methods for planning, particularly for distinguishing objects that appear similar but have different physical dynamics.
Temporal Structure: Capturing the full temporal dynamics of interactions, rather than just final outcomes, allows for precise planning using intermediate trajectory points.
Efficient Generalization: Behavioral similarity matching enables robust few-shot generalization to novel objects, outperforming methods reliant on geometric or visual similarity.

The authors acknowledge limitations, including the focus on single-object affordances, the reactive nature of drift-based replanning (which occurs post-execution), and the computational efficiency of A* compared to PDDL solvers. They suggest future work in inter-object relational symbols, pre-execution verification, and PDDL-compatible symbolic operator generation.