💻 computer science

Learning 3D Affordances for Blade Insertion in Cluttered Stowing

This paper introduces VulcanVoxel, a spatial inference framework that uses masked autoencoders on 3D occupancy fields to learn multi-modal blade insertion affordances in cluttered environments directly from unannotated real-world data, significantly outperforming traditional pose-based methods in both coverage and inference speed.

Original authors: Tianyu Li, Harpreet Sawhney, Minju Jung, Aditya Mehrotra, Kunal Mehrotra, Mudit Agrawal

Published 2026-07-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Tianyu Li, Harpreet Sawhney, Minju Jung, Aditya Mehrotra, Kunal Mehrotra, Mudit Agrawal

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to slide a long, rigid knife into a messy drawer full of clothes, toys, and random junk. Your goal isn't just to grab an item; it's to find a clear path to slide the knife in, push some things aside, and create space without breaking anything.

This is exactly what the robot in this paper is trying to do: insert a blade into a cluttered fabric bin to make room for new items. The researchers call this "blade insertion."

Here is the simple breakdown of their problem, their solution, and why it works, using everyday analogies.

The Problem: The "Wrong Question"

Most robots try to solve this by asking: "What is the exact angle and position (pose) the blade should be in?"

The paper argues this is like trying to solve a maze by only looking at the final exit coordinates. The problem is that in a messy drawer, there isn't just one correct way to slide the knife in. There might be three or four different gaps that work perfectly.

However, the robot's training data only shows it one successful attempt (the one a human or a previous robot actually did). If the robot tries to learn "the answer" from this single example, it gets stuck. It learns to copy that one specific move and fails to realize that other gaps in the mess would also work. It's like a student who memorizes the answer to one math problem but can't solve a similar one with slightly different numbers.

The Solution: "VulcanVoxel" (The 3D Puzzle Solver)

The team built a new system called VulcanVoxel. Instead of asking "Where is the blade?", they ask a different question: "Where is there empty space big enough for the blade to fit?"

Think of the messy bin as a giant 3D grid of tiny cubes (voxels).

The Old Way: The robot guesses a single 3D coordinate for the blade.
The VulcanVoxel Way: The robot looks at the whole grid and lights up every single cube where the blade could physically fit without hitting a wall or an object.

The Magic Trick (Masked Autoencoder):
To teach the robot to do this, the researchers used a clever game of "fill in the blanks."

They showed the robot a picture of the messy bin.
They covered up (masked) the part of the picture showing where the blade should go.
They asked the robot: "Based on the walls and the objects you can see, where could the blade fit?"

Because the robot has to figure out the geometry of the space itself (not just copy a specific move), it realizes that if Gap A works, Gap B might also work. This allows it to come up with multiple valid solutions (multi-modal predictions) even though it only saw one example during training.

The Results: Why It Matters

The researchers tested this on thousands of real warehouse scenarios.

The "Copycat" Robots: The best previous methods could only find a valid path about 71% of the time. If the robot's first guess was wrong, it had no backup plan.
VulcanVoxel: Because it understands the space rather than just the move, it found a valid path 89% of the time.
The "Backup Plan": When VulcanVoxel suggests 5 different places to put the blade, at least one of them is almost always a safe, valid spot. The old methods usually only offered one spot, and if that was wrong, the whole task failed.

The Speed Trick (The Student)

The main VulcanVoxel system is smart but a bit slow (it takes about 2.3 seconds to think). To make it fast enough for a real factory, they trained a "student" robot.

The Teacher: The slow, smart 3D grid thinker.
The Student: A faster version that looks at a simple 2D photo (RGB) and guesses the 3D answer.
The Result: The student is 46 times faster (30 milliseconds) and still gets about 65% of the teacher's accuracy. It's like a student who learned the principles of geometry from the teacher and can now solve problems instantly without needing to draw the whole 3D grid every time.

Summary

The paper claims that to teach robots how to navigate messy spaces, we shouldn't teach them specific moves (poses). Instead, we should teach them to understand the geometry of empty space. By treating the "blade insertion" as a 3D puzzle of "where does this shape fit?" rather than "where did we put it last time?", the robot becomes much more flexible, finds more solutions, and is less likely to crash into things.

They also released a massive dataset of these real-world blade insertion attempts so other researchers can try to solve the same puzzle.

Technical Summary: Learning 3D Affordances for Blade Insertion in Cluttered Stowing

Problem Statement
Robotic stowing involves inserting a rigid blade into densely cluttered fabric bins to sweep items aside and create space. This task requires reasoning about free-space affordances: identifying volumes of 3D space where a tool can safely navigate without colliding with surrounding geometry. Unlike surface contact affordances (where a tool touches an object), free-space affordances depend on the navigable volume constrained by bin walls and occluded items.

Standard approaches treat this as a regression or generative problem over SE(3) poses (6-DoF tool placements). However, production data is inherently unimodal: for any given bin state, the system executes only one strategy, even though multiple geometrically valid insertion paths often exist. Training generative models (e.g., flow matching) on this unimodal pose data causes them to collapse to a single mode, failing to recover the diverse set of valid geometric solutions. Furthermore, reasoning in pose space displaces the inference from the domain where geometric feasibility is naturally defined (3D occupancy), making it difficult to determine if unobserved placements are physically possible.

Methodology: VulcanVoxel
The authors propose VulcanVoxel, a framework that keeps inference in the spatial domain rather than projecting it into pose space. The core hypothesis is that blade affordances are spatial objects (subsets of 3D space) and should be inferred as such.

Representation: The system reconstructs the scene from a single RGB-D frame into a multi-channel 3D occupancy grid ( $32^3$ $3 2^{3}$ voxels). Channels encode:
- Combined wall and object occupancy ( $O_{occupy}$ ).
- Object-only occupancy ( $O_{object}$ ).
- Ground-truth blade occupancy ( $O_{blade}$ ).
Architecture: VulcanVoxel employs a 3D U-Net Masked Autoencoder (MAE).
- Training Objective: Instead of predicting a pose distribution, the model is trained to reconstruct the masked blade occupancy channel conditioned on the observed scene geometry.
- Channel Masking: During training, random combinations of the three channels are masked. This forces the model to learn inter-channel geometric relationships (e.g., inferring where a blade fits given the wall and object positions) rather than memorizing execution patterns.
- Loss Function: Due to the sparsity of blade voxels, the model uses a combination of Focal Loss and Dice Loss to handle class imbalance and enforce shape coherence.
Inference Pipeline:
- The model outputs a 3D probability field of valid blade occupancy.
- Pose Extraction: Valid connected components in the occupancy field are identified. Candidate SE(3) poses are enumerated around these components and scored based on volumetric overlap with the predicted field. This defers pose extraction to a post-hoc step, ensuring spatial reasoning occurs first.
Distillation: To address inference latency (the voxel-to-voxel model takes ~2.3s), a distilled student is trained to predict blade occupancy directly from RGB images in ~30ms. This student uses a ViT encoder and a 3D decoder, lifting 2D features to 3D via geometric constraints derived from camera calibration.

Key Contributions

Spatial Inference for Free-Space Affordances: The paper demonstrates that representing affordances as 3D occupancy fields, rather than SE(3) pose distributions, allows models to recover multi-modal predictions from unimodal production data.
Representational Analysis: Through systematic ablation (comparing input modalities, output representations, and training objectives), the authors show that:
- Operating in the spatial domain is necessary but not sufficient.
- The reconstruction objective (learning what is geometrically consistent) is the critical factor that enables multi-modality, distinguishing VulcanVoxel from flow-matching baselines operating in the same spatial domain.
Generalization to Failures: The model can identify geometrically valid free-space volumes even in bin configurations where the production system failed, effectively turning failure episodes into informative training signals.
StowAffordance Dataset: The release of a large-scale dataset containing 10,000+ real blade insertion cycles with RGB-D observations, SE(3) trajectories, and outcome labels.

Results
Evaluated on 1,000 real-world test episodes (July 2025) and additional temporal/high-clutter sets:

Coverage: VulcanVoxel achieves a Top-5 Coverage (Cov5) of 0.89, significantly outperforming the best pose-based baseline (Voxel FM Pose) at 0.71. This means in 89% of scenes, at least one of the five predicted poses is geometrically valid.
Multi-modality: VulcanVoxel predicts an average of 2.7 distinct modes per scene with a spatial spread ( $\sigma$ ) of 8.4 cm. In contrast, flow-matching baselines collapse to single modes ( $N_m \approx 1.0$ ) or produce geometrically incoherent spread.
Accuracy: While VulcanVoxel's single-prediction cost is slightly higher than the best pose-based baseline (0.81 vs. 0.60), it matches production performance on single-prediction accuracy while offering vastly superior diversity.
Distillation: The RGB student model achieves 30ms inference (46x speedup) while recovering ~65% of the teacher's prediction quality and maintaining partial multi-modality.
Failure Generalization: On production failure cycles, VulcanVoxel maintains high coverage (>0.80), suggesting many production failures stem from execution/force-feedback issues rather than perceptual errors.

Significance and Claims
The paper argues that tool affordances are spatial objects, and inferring them in SE(3) pose space discards the geometric structure essential for feasibility reasoning. By keeping inference in the native 3D domain and using a reconstruction objective, the model learns "where blades fit" as a local geometric condition rather than "where blades were inserted" as a global distribution.

The authors claim this approach is a canonical solution for free-space affordance learning in rigid tool manipulation. They posit that this principle extends beyond stowing to any manipulation domain where geometric compatibility is the primary constraint, suggesting that spatial inference should be the default approach when production data is abundant, even if that data is unimodal. The work highlights that the bottleneck in such tasks is often the domain of inference (pose vs. space) and the training objective (generative vs. reconstruction), rather than model capacity or data scale.