💻 computer science

Mash, Spread, Slice! Learning to Manipulate Object States via Visual Spatial Progress

The paper introduces SPARTA, a unified framework that leverages visual spatial progress segmentation to enable robots to effectively learn and execute manipulation tasks involving continuous object state changes, such as mashing and slicing, through structured observations and dense rewards.

Original authors: Priyanka Mandikal, Jiaheng Hu, Shivin Dass, Sagnik Majumder, Roberto Martín-Martín, Kristen Grauman

Published 2026-03-23

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Priyanka Mandikal, Jiaheng Hu, Shivin Dass, Sagnik Majumder, Roberto Martín-Martín, Kristen Grauman

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to cook. Most robots today are great at moving things around: picking up a cup, putting it on a table, or opening a door. These are like moving rigid blocks; the object stays the same, it just changes location.

But what about changing the object itself?

Mashing a banana into smooth mush.
Spreading peanut butter evenly on toast.
Slicing a cucumber into thin rounds.

These tasks are tricky. The object doesn't just move; it morphs. It changes shape, texture, and color right under the robot's nose. If you ask a standard robot to "mash this banana," it might just push the whole banana off the table because it doesn't understand that it needs to press down on the unmashed parts and avoid the already-mashed parts.

This paper introduces SPARTA, a new way to teach robots how to handle these "squishy, changing" tasks.

The Core Idea: The "Painting" Analogy

Think of the robot's job like painting a wall.

The Goal: Cover the whole wall with paint.
The Problem: If you just stare at the wall, you see a mix of white (unpainted) and blue (painted). It's hard to tell exactly where you are or how much progress you've made just by looking at the messy colors.
The SPARTA Solution: Instead of looking at the messy wall, SPARTA gives the robot a magic map.
- Red areas on the map mean: "This part is still raw/unmashed/unpainted. Go here!"
- Green areas mean: "This part is done. Don't waste time here."

This "magic map" is called a SPOC map (Spatially Progressing Object State Change). It strips away the confusing details (like the specific color of the banana or the texture of the bread) and focuses only on what needs to be done next.

How SPARTA Works: Two Modes

The paper shows that SPARTA can work in two different "personalities" depending on the job:

1. SPARTA-L (The Learner) 🧠

Best for: Tricky jobs that need precision, like spreading sauce with a thin brush or slicing a cucumber.
How it works: This is like a student taking a test. The robot tries an action, looks at the magic map, and gets a score.
- Did you mash a new chunk? Good job! (+1 point).
- Did you mash the same spot again? No points.
- Did you miss the whole banana? Zero points.
Because the robot gets a score for every tiny step of progress (not just a "pass/fail" at the very end), it learns fast. In the experiments, it learned to mash bananas and spread jam in just 1.5 to 3 hours of real-world practice, with no human showing it how to do it.

2. SPARTA-G (The Greedy Go-Getter) 🏃

Best for: Big, simple jobs where you have a wide tool, like smashing potatoes with a big masher.
How it works: This robot doesn't "learn" over time. It just looks at the magic map and asks, "Where is the biggest pile of red (unmashed) stuff right next to me?" It moves there immediately.
It's fast, lightweight, and doesn't need training. It's like a vacuum cleaner that just follows the dirt.

Why Is This a Big Deal?

Before this, robots struggled with these tasks because:

They got confused by looks: A mashed banana looks different than a sliced cucumber. Standard robots tried to memorize every specific look, which is impossible. SPARTA ignores the looks and focuses on the state (mashed vs. unmashed).
They got lost without a map: If you tell a robot "mash the banana" and only give it a reward at the very end when it's done, the robot is flying blind. It's like trying to find a needle in a haystack without a magnet. SPARTA gives the robot a magnet (the dense reward) that guides it step-by-step.

The Results

The team tested SPARTA on 10 different objects (bananas, avocados, bread, cucumbers, etc.) with three different tasks (mashing, spreading, slicing).

Old methods (like just guessing or using simple "did you finish?" rewards) failed miserably.
SPARTA succeeded, turning messy, uncooked ingredients into perfectly processed food.

The Takeaway

SPARTA is like giving a robot a pair of X-ray glasses that can see "what is done" and "what is left to do." By focusing on the progress rather than the specific appearance of the object, robots can finally learn to do the messy, squishy, real-world chores that humans do every day, without needing a human to hold their hand and show them how.

In short: SPARTA teaches robots to stop just moving objects around and start actually cooking with them.

1. Problem Definition

The paper addresses a gap in robotic manipulation: while most research focuses on rigid-body motion (picking, placing, rotating), many real-world tasks involve Object State Changes (OSC). In OSC tasks, an object's physical and visual state evolves progressively (e.g., mashing a banana, spreading jam, slicing a cucumber) without necessarily changing the object's global pose.

Key Challenges:

Representation: Raw RGB observations entangle object appearance with state, making it difficult to distinguish between "actionable" (untransformed) and "transformed" regions. This hinders generalization across different objects.
Reward Design: Traditional sparse rewards (success/failure at the end) provide insufficient guidance for exploration. Goal-conditioned rewards often rely on global embeddings that miss fine-grained, incremental progress.
Control: The robot must reason about where transformations have occurred and where they are still needed, requiring continuous, fine-grained interaction rather than discrete pose changes.

2. Methodology: SPARTA Framework

The authors propose SPARTA (Spatial Progress-Aware Robotic object TransformAtion), a unified framework that leverages Spatially Progressing Object State Change (SPOC) segmentation maps to guide robot learning.

A. Visual Affordance Generation (SPOC)

Instead of using raw RGB frames, SPARTA uses a vision pipeline to generate SPOC maps (binary segmentation masks) that distinguish between:

Actionable Regions: Parts of the object that still need transformation (e.g., unmashed potato).
Transformed Regions: Parts already processed (e.g., mashed potato).
Background.

Pipeline:

Object Detection: Uses Grounded-SAM to extract the object mask.
Region Sampling: Samples intra-object regions using farthest-point sampling.
Classification: Uses GPT-4o (a Vision-Language Model) to classify sampled regions as "actionable" or "transformed" based on color-coded overlays.
Tracking: Uses DeAOT to propagate these masks across frames efficiently (~0.2s/frame), avoiding the latency of per-frame LLM queries.

Key Insight: This decouples visual perception from policy learning, providing a structured, appearance-invariant representation of progress.

B. Policy Variants

SPARTA offers two policy types within the same Markov Decision Process (POMDP) framework:

SPARTA-L (Reinforcement Learning):
- Goal: Learn a policy from scratch in the real world without demonstrations or simulation.
- Reward Function: Uses a dense, spatially grounded reward ( $R_{spoc}$ ) calculated as the ratio of newly transformed area to actionable area at each step.
- Algorithm: Uses Soft Actor-Critic (SAC) with regularization. The dense reward stabilizes exploration, allowing the agent to learn sample-efficiently.
SPARTA-G (Greedy Control):
- Goal: Fast, lightweight deployment for tasks with symmetric tools (e.g., mashing) where precise directional control is less critical.
- Mechanism: A non-parametric controller that evaluates 8 discrete directions around the end-effector and selects the one with the highest density of "actionable" pixels in the SPOC map.
- Advantage: Requires no training; acts as a reactive heuristic.

3. Key Contributions

Unified Framework for OSC: The first system to unify object state change manipulation under a single visual progress representation, supporting both learning-based and greedy control.
Progress-Aware Visual Abstraction: Introduces the use of SPOC maps to strip away appearance variability, providing the robot with a clear signal of "what has changed" vs. "what remains."
Dense, Interpretable Rewards: Demonstrates that visual progress signals can replace sparse rewards, enabling sample-efficient real-world RL without human demonstrations or simulators.
Real-World Validation: Successfully trains policies directly on a real robot (Franka Emika Panda) for complex, non-rigid tasks.

4. Experimental Results

Setup:

Tasks: Spreading (syrup on donut), Mashing (banana/avocado), Slicing (cucumber/butter).
Objects: 10 diverse real-world objects with varying shapes, textures, and colors.
Baselines: Random control, Sparse rewards (binary success), and LIV (state-of-the-art goal-conditioned representation learning).

Findings:

Performance: SPARTA (both L and G variants) significantly outperformed all baselines.
- SPARTA-L achieved 60–78% transformation coverage across tasks, whereas baselines (Sparse, LIV) struggled to exceed 15–20%.
- SPARTA-G was highly effective for mashing (63–75% coverage) due to the robustness of large tools against perceptual noise.
Sample Efficiency: SPARTA-L learned usable policies in 1.5 to 3 hours of real-world training with zero human demonstrations. Baselines failed to converge or learned degenerate behaviors.
Reward Stability: SPARTA's dense rewards produced smooth, monotonic learning curves. In contrast, LIV rewards were unstable and failed to capture incremental progress.
Generalization: The system generalized to unseen objects (e.g., trained on bread, tested on cheese) because the policy relies on spatial progress rather than specific object textures.

5. Significance and Future Work

Significance:

Beyond Rigid Bodies: Moves the field of robotics beyond pick-and-place to handle deformable, non-rigid objects where the goal is a state change rather than a pose change.
No Sim-to-Real Gap: By learning directly in the real world using visual progress signals, the method avoids the difficulties of simulating complex contact dynamics and material deformations.
Versatility: Proves that a single visual representation (SPOC maps) can power both simple greedy controllers and complex RL agents.

Limitations & Future Directions:

Geometry Generalization: Performance degrades on significantly different geometries (e.g., policies trained on rectangular slices struggle with circular tortillas).
Occlusion: The current pipeline requires the end-effector to lift between actions to capture visual inputs, preventing continuous perception during contact. Future work aims to develop occlusion-resilient reasoning.

In conclusion, SPARTA establishes progress-aware visual representations as a foundational tool for enabling robots to perform complex, everyday manipulation tasks that require continuous state transformation.