💻 computer science

SIR: Structured Image Representations for Explainable Robot Learning

The paper introduces Structured Image Representations (SIR), a method that utilizes learnable, sparse scene graphs as intermediate representations to enhance the performance and intrinsic explainability of robot policies, enabling the detection of dataset biases through graph analysis.

Original authors: Paul Mattes, Jan Schwab, Jens Bosch, Nils Blank, Maximilian Xiling Li, Minh-Trung Tang, Moritz Haberland, Rudolf Lioutikov

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Paul Mattes, Jan Schwab, Jens Bosch, Nils Blank, Maximilian Xiling Li, Minh-Trung Tang, Moritz Haberland, Rudolf Lioutikov

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Black Box" Robot

Imagine you teach a robot to make a sandwich by showing it videos of humans doing it. The robot learns to move its arm, but it does so by looking at a giant, blurry wall of pixels (an image).

The problem is that the robot's "brain" (its policy) is a black box. It sees the pixels, makes a decision, and moves. But if you ask, "Why did you grab the knife instead of the spoon?" the robot can't answer. It just knows that "pixels looking like this" leads to "grabbing the knife."

Furthermore, if you put a random toy on the counter (a distraction), the robot might get confused and try to grab the toy instead of the knife. Because it's looking at the whole messy picture, it can't easily tell what matters and what doesn't.

The Solution: SIR (The "Smart Organizer")

The authors propose a new method called SIR (Structured Image Representations). Instead of letting the robot look at the messy wall of pixels, SIR forces the robot to organize the scene first, like a smart organizer or a project manager.

Here is how it works, step-by-step:

1. The Initial List (The Fully Connected Graph)

First, the robot looks at the image and identifies every object it sees: the fridge, the door, the cup, the robot's own hand, and even the wall.

Analogy: Imagine the robot writes down a list of everyone in a crowded room. It connects every person to every other person on a giant whiteboard. This is called a "Fully Connected Graph." It's accurate, but it's overwhelming and messy.

2. The Filter (Sparsification)

This is the magic part. The robot has a "manager" module that looks at this giant list and asks: "For the specific task of 'Open the Microwave,' which of these people actually matter?"

The manager deletes the people who don't matter (like the wall, the floor, or a random toy).
It keeps only the essential connections (the Robot Hand, the Microwave, and maybe the Door handle).
Analogy: The manager takes a red marker and crosses out 90% of the names on the list, leaving only the 3 or 4 people who are actually involved in the conversation. This creates a Sparse Graph.

3. The Action (Decision Making)

Now, the robot only looks at this tiny, clean list of important items to decide what to do next.

Why this is better: Because the robot is only looking at the "important" items, it is much harder for it to get distracted by random toys. It also makes the robot's decision process transparent.

Why This Matters: The "X-Ray" Vision

The paper claims that SIR isn't just about making the robot work better; it's about understanding why it works (or fails).

Because the robot explicitly decides which objects to keep and which to throw away, we can look at that decision and say:

"Ah, I see! The robot kept the 'Microwave' and the 'Hand', so it knows what to do."
"Wait, the robot kept the 'Wall' and the 'Toy' but threw away the 'Microwave'. That's weird!"

The Discovery:
When the researchers looked at the robot's "filtered lists" (the sparse graphs), they found some funny mistakes that would have been impossible to see with the old "black box" method:

Spurious Correlations: In some cases, the robot learned that if it saw a specific type of window, it should open the drawer. It didn't care about the drawer handle; it just cared about the window. The "filter" showed the robot was cheating by looking at the wrong clues.
Positional Bias: In another case, the robot learned to just move its arm in a fixed circle because, in the training videos, the robot always started in that position. The "filter" showed the robot completely ignored the actual door it was supposed to open!

The Results: Better and Stronger

The researchers tested this on a robot learning kitchen tasks (like opening doors or drawers).

Success Rate: The new SIR method was more successful (about 19.5% success) than the old image-based method (14.8%).
Distraction Test: When they added random distracting objects to the scene, the old robot got confused and failed more often. The SIR robot barely noticed the distractions and kept working.

Summary Analogy

Old Method: A student trying to solve a math problem by staring at a messy whiteboard covered in random scribbles, numbers, and doodles. They guess the answer based on the whole mess. If someone draws a new doodle, they get confused.
SIR Method: The student first writes down only the specific numbers and variables needed for the equation, crossing out all the doodles. They solve the problem using only the clean list. If they get the wrong answer, you can look at their list and immediately see, "Oh, you forgot to include the 'X' variable!" or "You included the doodle of a cat instead of the number 5!"

In short: SIR makes robots smarter, less easily distracted, and, most importantly, honest about what they are actually looking at when they make a decision.

Technical Summary: Structured Image Representations for Explainable Robot Learning (SIR)

Problem Statement

Existing robot policies based on learned visual embeddings often lack explicit structure and are sensitive to visual distractions. While these embeddings provide compact encodings of visual information, they remain opaque, rendering the decision-making process difficult to interpret. Furthermore, as real-world robot agents operate in complex environments with numerous observed objects, there is a growing need for structured, expressive scene representations that facilitate both performance and explainability in Goal-Conditioned Imitation Learning (GCIL). Current approaches often rely on unstructured visual features or require complex planning algorithms, limiting their applicability to step-wise learning without explicit key-points.

Methodology: Structured Image Representations (SIR)

The authors propose Structured Image Representations (SIR), a framework that leverages Scene Graphs (SGs) as an intermediate representation for robot policy learning. Unlike post-hoc explainability methods, SIR is intrinsically explainable because the sparse graph serves as a learned intermediate state during action generation.

The architecture consists of four core components:

Scene Graph Generation:
- The system constructs an initial Fully-Connected Graph (FC-Graph) where nodes represent objects detected in the scene (via ground-truth or predicted segmentation masks).
- Initial node representations are derived from diverse image modalities, including:
  - Label: One-hot encoded object labels.
  - Cropped-Image-Feature: Visual embeddings from pre-trained networks (e.g., ResNet18) applied to object bounding box crops.
  - BB-Coordinates: Normalized 2D bounding box coordinates.
  - Point-Cloud-Feature: Embeddings from pre-trained networks (e.g., PointNet) applied to object point clouds.
- Edge features are initialized based on geometric distances or set to 1.
Learnable Graph Sparsification:
- A trainable module, termed FiLMDecoder (a two-layer Transformer-Decoder with Adaptive Layer Normalization conditioned on language goals), predicts a score for every node.
- The system employs instruction-grounded node selection to retain only the top- $k$ highest-scoring nodes, creating a minimal, task-relevant sub-graph.
- To ensure the sparsification is learnable and differentiable, the authors introduce:
  - A soft histogram loss to encourage uniform distribution of node scores and prevent collapse.
  - An L1 loss on node weights to guide the selection of instruction-relevant nodes.
  - A mechanism where node weights ($NW$) are incorporated into edge weights and the pooling step. This ensures that information from "removed" nodes does not propagate into the final graph embedding, making the sparsification truly end-to-end.
Scene Graph Embedding Generation:
- The sparse sub-graph is processed by a two-layer Graph Attention Network (GATv2) with residual connections.
- A global average pooling layer aggregates the features of the retained nodes to produce the final scene representation (graph embedding).
Action Generation:
- The resulting graph embedding, combined with an embedded language goal (via CLIP), serves as the state input for downstream action generators. The authors evaluate two generators: the Multimodal Diffusion Transformer (MDT) and a Behavior Cloning (BC)-Transformer.

Key Contributions

Investigation of SGs in GCIL: The paper provides a thorough investigation into using Scene Graphs as direct state representations for robot learning, systematically analyzing which image modalities (symbolic, geometric, visual) serve as effective initial node features.
Learnable Sparsification: The authors propose a novel method for end-to-end learnable SG sparsification. This creates a sparse sub-graph that acts as a direct explanation of the model's decision-making process, revealing which objects and interactions are deemed critical.
Intrinsic Explainability: By analyzing deviations between the learned sub-graphs and human expectations (e.g., inclusion of distractors or omission of key objects), the method allows for the detection of dataset biases, such as spurious correlations and positional biases.

Experimental Results

The method was evaluated on the RoboCasa and CALVIN benchmarks using 100 rollouts per configuration.

Performance: On RoboCasa, SIR achieved an average success rate of 19.5%, outperforming image-based baselines (14.81%) and fully-connected graph models without sparsification (16.98%).
Modality Integration: Graph-based models demonstrated superior ability to integrate diverse modalities. For instance, while point-cloud-only baselines performed poorly (4.13%), graph-based models utilizing point-cloud features achieved significantly higher success rates (11.08% to 15.04%).
Robustness to Distractors: When novel distractor objects were introduced during inference, image-based baselines suffered a performance drop of ~3.3%. In contrast, SIR and other graph-based models showed almost no performance degradation, demonstrating high robustness to visual distractions.
Ablation Studies: The study confirmed that instruction-grounded sparsification with soft histogram loss yields the best performance compared to random removal, naive removal, or threshold-based methods.

Significance and Claims

The paper claims that SIR offers a significant advancement in explainable robot learning by moving beyond opaque visual embeddings to structured, relational representations.

Intrinsic vs. Post-hoc: The authors emphasize that SIR is not a post-hoc explanation method (like using a VLM to filter objects after training). Instead, the sparse graph is a learned intermediate representation. This allows the model to reveal its actual decision-making logic, including potential flaws.
Bias Detection: A critical finding is that analyzing the learned sub-graphs can uncover dataset biases. For example, the authors observed that in certain tasks, the model successfully executed actions by ignoring the target object and relying solely on the robot's own state (positional bias), or by exploiting spurious correlations with irrelevant background objects.
Interpretability: The ability to visualize the "task-explanation" sub-graphs provides direct insight into whether the model is succeeding for the right reasons or exploiting shortcuts in the training data.

The authors conclude that while SIR improves performance and robustness, its most valuable contribution is the ability to use the learned sparse graphs as a debugging tool for both model behavior and dataset quality.