💻 computer science

SPARC: Reliable Spatial Annotations from Robot Demonstrations at Scale

This paper introduces SPARC, a risk-aware framework that automatically generates high-quality, reliability-calibrated spatial annotations from large-scale robot demonstrations, significantly improving object grounding and policy performance in complex real-world scenes compared to existing automated methods.

Original authors: Nils Blank, Paul Mattes, Maximilian Xiling Li, Jakub Suliga, Thomas Roth, Moritz Reuss, Pankhuri Vanjani, Rudolf Lioutikov

Published 2026-06-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Nils Blank, Paul Mattes, Maximilian Xiling Li, Jakub Suliga, Thomas Roth, Moritz Reuss, Pankhuri Vanjani, Rudolf Lioutikov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to cook by showing it videos of humans making toast. You want the robot to learn exactly which piece of toast is being moved, where it starts, and where it ends up.

The problem is, if you just ask a computer to "watch" these videos and guess what's happening, it often gets confused. It might see a shiny toaster, think that's the object being moved, and confidently label it as such—even though the robot is actually holding the bread. This is like a student guessing the answer on a test because they recognize a word, but getting the whole question wrong.

This paper introduces a new system called SPARC (Spatial Annotations from Robot Demonstrations with Reliability Calibration). Think of SPARC as a super-smart, risk-aware teaching assistant that watches robot videos and labels them with extreme precision.

Here is how it works, broken down into simple concepts:

1. The Problem: The "Confident but Wrong" Trap

Existing automated systems try to label robot videos by finding objects and tracking them. However, they rely on a "confidence score" that basically asks, "Does this look like a toaster?"

The Flaw: In a messy kitchen, a shiny toaster might look more like a toaster than the actual piece of toast being held. The system gets 99% "confident" it's the toaster, but it's actually labeling the wrong object.
The Result: Researchers have to choose between using lots of noisy, wrong labels or throwing away most of their data to keep only the few "safe" ones.

2. The Solution: SPARC's "Detective Work"

SPARC doesn't just ask, "What does this look like?" It asks, "What is the robot actually touching and moving?"

It acts like a detective using three specific clues to figure out what is really happening:

Clue 1: The "When" (Timing): It watches the robot's hand (gripper). It knows exactly when the hand closes, when it holds, and when it lets go. It only pays attention to objects that move during that specific "holding" time.
Clue 2: The "Where" (Proximity): It checks if the object is physically close to the robot's hand. If an object is far away, it's probably not the one being manipulated, even if it looks similar.
Clue 3: The "Who" (Filtering): It knows what the robot's own arm looks like. If the system sees a box that looks like the robot's arm, it ignores it.

3. The "Reliability Score": A Trust Meter

Instead of just saying "This is the toast," SPARC gives every label a Trust Score (from 0 to 1).

If the object moved exactly when the hand closed, was right next to the hand, and wasn't the robot itself, it gets a high score (e.g., 0.95).
If the object was far away or didn't move when the hand closed, it gets a low score.

This allows researchers to set a "trust threshold." They can say, "I only want labels with a trust score above 0.9." Because SPARC is so good at filtering out the bad guesses, they can keep three times more useful data than previous methods while still being highly accurate.

4. The Results: Better Robots, Faster

The authors tested this on 1,700 human-annotated videos (the "gold standard" ground truth) to see if SPARC could match human accuracy.

Accuracy: SPARC correctly identified the manipulated object 80% of the time at high confidence, compared to only 58% for standard methods.
Efficiency: It is 24 times faster than a human annotator.
Real-World Impact: When they used SPARC's labels to train new robot brains (AI models), those robots became much better at picking up objects in messy, cluttered rooms. They succeeded 64% of the time, compared to only 21% for robots trained on the older, noisier data.

5. The "IA-Bench" (The Final Exam)

To prove their system works, the authors created a new test called IA-Bench (Interaction-Aware Bench).

Imagine a test where you have to watch a video and point to exactly which object the robot touched.
Old tests only looked at single frames (like a snapshot). SPARC's test looks at the whole video and the robot's hand movements.
SPARC aced this test, proving that looking at the interaction (the touch and move) is the key to understanding robot tasks.

Summary

SPARC is a system that stops robots from guessing based on "what things look like" and starts them guessing based on "what things actually do." By combining the robot's hand movements with the video, it creates a massive library of perfectly labeled training data. This library helps train robots to be smarter, more accurate, and much faster learners, all without needing a human to sit there and check every single video.

Technical Summary: SPARC – Reliable Spatial Annotations from Robot Demonstrations at Scale

1. Problem Statement

Generating high-quality, structured spatial annotations (e.g., bounding boxes, object trajectories, manipulation phase labels) from robot demonstrations at scale is a critical bottleneck for training grounded robot policies and embodied foundation models. Existing automated pipelines typically rely on chaining object detectors with trackers. However, these approaches suffer from a fundamental scale-quality tradeoff:

Misaligned Confidence: Detector confidence measures recognition likelihood, not whether the detected object is the one actually being manipulated. In cluttered scenes, pipelines often lock onto plausible but incorrect distractors with high confidence.
Ineffective Filtering: Relying on detector confidence or simple motion magnitude heuristics forces a choice between accepting noisy labels or discarding a vast majority of useful samples to maintain precision.
Cost of Human Review: While human annotation is reliable, it is prohibitively expensive for large-scale datasets. Simulation offers ground truth but struggles with the reality gap.

Consequently, there is a lack of automated methods that can generate spatio-temporal annotations with a reliable, per-annotation quality signal derived from physical interaction evidence rather than just visual appearance.

2. Methodology: The SPARC Framework

The authors introduce SPARC (Spatial Annotations from Robot Demonstrations with Reliability Calibration), a risk-aware framework that automates the labeling of robot demonstrations while assigning a reliability score to each annotation. The pipeline operates in three stages:

Stage 1: Subtask Decomposition and Phase Detection

Interaction Segmentation: Given a long-horizon demonstration (visual frames, proprioceptive states, and language instructions), SPARC decomposes the trajectory into short, object-centric segments.
Gripper Phase Extraction: Using the normalized gripper signal, the system identifies grasp cycles (approach, interact, release) via a dual-pass hysteresis procedure to filter out noise and transient slips.
Language-Grounded Parsing: A large language model (Qwen3.6-30B) aligns the detected grasp cycles with the language instruction to generate structured subtask records, identifying the manipulated object, action type, and start/target locations.

Stage 2: Candidate Proposal and Tracking

Candidate Generation: At a keyframe chosen to minimize occlusion, an open-vocabulary detector (LLMDet) with a low threshold proposes object candidates. Instance masks are generated using SAM2.
Dense Tracking & 3D Lifting: The system applies a dense tracker (AllTracker) to generate pixel-level tracks for each candidate. These 2D tracks are lifted to 3D using a monocular geometry estimator (MoGe-2), creating a spatio-temporal representation of each candidate's motion.

Stage 3: Reliability Scoring

Instead of relying on detector confidence, SPARC computes a Composite Annotation Reliability Score ( $R_i$ ) based on three physically grounded cues:

Phase-Aware Motion ( $A_i$ ): Measures if the candidate moves significantly during the interaction phase compared to non-interaction phases. This suppresses background motion and tracking jitter.
3D Gripper Proximity ( $P_i$ ): Calculates the fraction of the candidate's 3D points that fall within an adaptive sphere around the gripper during manipulation.
Robot-Overlap Penalty ( $\Pi_i$ ): A soft quadratic penalty that suppresses candidates overlapping with the robot's own geometry (e.g., the gripper itself), while preserving true objects that are in contact with the gripper.

The final score combines these signals with the detector confidence ( $D_i$ ), using an adaptive weight that reduces reliance on appearance when 3D proximity evidence is strong. A threshold on this score allows users to explicitly trade off annotation quality against dataset scale without human review.

3. Key Contributions

Reframing Annotation as Selective Prediction: The authors propose treating automatic spatio-temporal annotation as a selective prediction problem, introducing a per-annotation reliability score grounded in physical interaction evidence rather than detector confidence.
IA-Bench Benchmark: The introduction of Interaction-Aware Bench (IA-Bench), a novel benchmark comprising 1,748 human-annotated start and target locations for manipulated objects across 12 diverse embodiments (including bimanual setups) and settings. Unlike static benchmarks, IA-Bench evaluates the ability to ground objects based on video and proprioception.
Large-Scale Dataset & Performance: The generation of a high-quality dataset of 167k annotated trajectories (from 280k raw trajectories) by filtering for 95% precision. The authors demonstrate that models fine-tuned on this data achieve state-of-the-art results on object-grounding and pointing benchmarks, outperforming similarly sized models trained on human-annotated data.

4. Experimental Results

Annotation Accuracy: On IA-Bench, SPARC achieves 80.2% interacted object localization accuracy at high confidence, significantly outperforming a detection-only baseline (58.1%) and other motion-based heuristics.
Data Retention: At a 95% precision target, SPARC retains 58% of samples, whereas the strongest trajectory-filtering baseline retains only 20%. This demonstrates SPARC's ability to maintain high coverage without sacrificing quality.
Downstream VLM Performance: Models (Qwen3.5-4B) fine-tuned on SPARC-annotated data outperform baselines on diverse spatial pointing tasks (vacant-space, affordance, trace prediction) and achieve state-of-the-art results on benchmarks like Where2Place, RefSpatial, and RoboRefIt.
Real-World Policy Robustness: In a real-world cluttered tabletop setting, policies trained on SPARC-annotated data with reasoning traces achieved a 31% success rate, more than doubling the performance of detection-annotated baselines (12%) and tripling the no-reasoning baseline (9%).

5. Significance and Claims

The paper claims that SPARC addresses the critical "scale-quality" bottleneck in robot learning by leveraging the inherent spatio-temporal structure of robot tasks. By shifting from appearance-based confidence to interaction-aware reliability, SPARC enables the creation of large-scale, high-quality datasets without human verification.

The authors assert that:

Reliability is Calibrated: The proposed score effectively estimates correctness, allowing for controllable quality-coverage tradeoffs.
Auto-Annotation Can Surpass Human Data: Despite being fully automatic and unverified by humans, SPARC-generated annotations yield models that match or exceed the performance of models trained on human-annotated data (e.g., EO-1.5M) on embodied grounding tasks.
Reasoning Benefits: High-quality spatial annotations are essential for training reasoning policies that can handle visually ambiguous and cluttered real-world scenes.

The work concludes that while SPARC is bounded by the capabilities of its underlying foundation models (e.g., struggles with transparent surfaces or identity switches in visually similar objects), it provides a robust, scalable pathway for generating the structured spatial supervision required for the next generation of embodied AI.