💻 computer science

Affostruction: 3D Affordance Grounding with Generative Reconstruction

This paper introduces Affostruction, a generative framework that reconstructs complete 3D object geometry from partial RGBD observations to ground affordances on both visible and unobserved surfaces, significantly outperforming existing methods through sparse voxel fusion, flow-based ambiguity modeling, and active view selection.

Original authors: Chunghyun Park, Seunghyeon Lee, Minsu Cho

Published 2026-04-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Chunghyun Park, Seunghyeon Lee, Minsu Cho

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a robot trying to pick up a coffee mug, but you can only see the front of it. You know there's a handle on the side, but your camera can't see it because it's hidden behind the mug's body. If you try to grab it based only on what you see, you might miss the handle entirely.

This is the problem Affostruction solves. It's a new AI system that helps robots "imagine" the whole object, not just the parts they can see, and figure out exactly where to touch, grab, or interact with it.

Here is how it works, broken down into three simple steps using everyday analogies:

1. The "Imagination Engine" (Generative Reconstruction)

The Problem: Most robots are like people with their eyes closed; they only know what is directly in front of them. If they see a mug from the front, they think the mug is just a flat circle. They don't know the handle exists.

The Solution: Affostruction is like a skilled sculptor who can finish a statue from a single fragment.

You show the robot a partial photo (like a puzzle piece).
Instead of just guessing, the AI uses a massive library of 3D shapes it has learned from millions of objects (a "foundation model").
It says, "I know what a mug looks like. Even though I can't see the handle, I know it must be there."
It builds a complete, invisible 3D model of the object in its mind, filling in all the missing parts (the back, the handle, the bottom) with high precision.

2. The "Intuitive Map" (Flow-Based Affordance Grounding)

The Problem: Once the robot has the full 3D model, it needs to know where to grab. But "grabbing" isn't always just one spot. You could grab a mug by the handle, by the rim, or even by the side if you are careful. This is called ambiguity—there isn't just one right answer.

The Solution: Instead of forcing the robot to pick one single spot (like a GPS pin), Affostruction creates a heat map.

Think of it like a weather map showing rain probability. Instead of saying "It will rain here," it says, "There is a 90% chance of rain here, and a 60% chance there."
The AI generates a "probability cloud" over the 3D object. It highlights all the valid places to interact.
This is crucial because it teaches the robot that there are many ways to interact with an object, making it more flexible and human-like in its movements.

3. The "Smart Detective" (Active View Selection)

The Problem: Sometimes the robot's first guess is wrong, or the object is too hidden. If the robot just spins around randomly, it wastes time.

The Solution: Affostruction acts like a detective who knows exactly where to look next.

After the robot makes its first guess about where the "handle" might be, it asks: "Where is the best place to stand to confirm this?"
It calculates the perfect new angle to move its camera to, specifically to see the hidden functional parts.
It's like playing a game of "Hot and Cold." Instead of walking in circles, the robot takes one smart step closer to the "hot" spot (the handle), gets a better look, and instantly refines its understanding.

Why is this a big deal?

Speed: It does all this incredibly fast (in about 7 seconds), which is vital for real-time robots.
Accuracy: It beats previous methods by a huge margin because it doesn't just guess; it reconstructs the missing geometry and generates multiple possibilities for interaction.
Real-World Use: This means robots won't just be able to "see" objects; they will be able to understand them. They can pick up a weirdly shaped tool, open a jar, or assemble furniture, even if they've never seen that specific object before, simply by imagining the full shape and figuring out the best way to touch it.

In short: Affostruction gives robots the superpower of imagination. It lets them fill in the blanks of the world around them and figure out the best way to interact with things, turning a robot that just "sees" into a robot that truly "understands."

1. Problem Statement

The paper addresses the challenge of 3D affordance grounding from partial RGB-D observations.

Context: Robots typically observe objects from limited viewpoints, resulting in significant occlusions. To manipulate an object (e.g., grasping a mug), a robot must understand functional regions (affordances) not just on visible surfaces but also on occluded parts (e.g., the handle).
Limitations of Existing Methods:
- Affordance Grounding Methods: Most existing open-vocabulary methods operate directly on partial point clouds, predicting affordances only on visible surfaces. They fail to reason about occluded functional regions.
- Reconstruction/Generation Methods: Multi-view reconstruction methods recover only observed surfaces, while 3D generation methods (often based on single RGB images) can complete unseen geometry but lack depth conditioning and camera extrinsics, leading to misaligned structures. They also do not predict functional properties.
Core Challenge: The system needs to simultaneously reconstruct complete 3D geometry (including unobserved regions) from partial RGB-D inputs and ground affordances on this complete shape, handling the inherent ambiguity of functional interactions (e.g., multiple valid grasp points).

2. Methodology: Affostruction

The authors propose Affostruction, a generative framework built upon the TRELLIS foundation model. It operates in three main stages:

A. Generative Multi-View Reconstruction

Sparse Voxel Fusion: Instead of processing views independently, the method aggregates features from multiple RGB-D views into a sparse voxel representation.
- Process: DINOv2 features are extracted from RGB images and projected into 3D space using depth maps ( $D$ ) and camera parameters ( $K, T$ ).
- Fusion: Features for overlapping voxels are averaged; non-overlapping voxels are united. This creates a constant-size representation ( $O(1)$ complexity) regardless of the number of input views.
- Conditioning: The fused features are combined with 3D positional encodings and fed into a Flow Transformer.
Stochastic Multi-View Training: To ensure the model leverages additional views effectively, it is trained with a stochastic strategy, randomly sampling 1–8 views per iteration. This prevents performance degradation when more views are provided at inference.
Output: The model extrapolates complete 3D geometry (occupied voxel positions) from partial observations, correcting for orientation and scale using depth conditioning.

B. Flow-Based Affordance Grounding

Generative Approach: Unlike discriminative models that predict a single "best" mask, Affostruction treats affordance prediction as a distribution modeling problem.
Flow Matching: A second Sparse Flow Transformer is trained to generate affordance heatmaps conditioned on natural language queries (encoded via CLIP).
Handling Ambiguity: By using rectified flow matching, the model learns the multi-modal distribution of valid interaction regions (e.g., multiple possible grasp locations).
Loss Function: It employs a combination of Binary Cross-Entropy (BCE) and Dice Loss to optimize for both point-wise accuracy and region-level overlap, specifically targeting binary occupancy patterns of functional regions.

C. Affordance-Driven Active View Selection

Closed-Loop Strategy: The predicted affordance heatmaps on the reconstructed geometry guide the selection of the "next-best" viewpoint.
Mechanism:
1. The reconstructed geometry is rendered with the affordance heatmap.
2. A set of candidate camera poses is evaluated based on an Affordance Visibility Metric (sum of heatmap values visible from that pose).
3. The viewpoint maximizing the visibility of high-affordance regions is selected.
Benefit: This iterative process prioritizes functional areas, progressively refining both the 3D reconstruction and the affordance grounding with limited view budgets.

3. Key Contributions

Generative Multi-View Reconstruction: Introduces a sparse voxel fusion mechanism that fuses depth-conditioned DINOv2 features from multiple views into a constant-complexity representation. This enables the extrapolation of complete geometry from partial RGB-D observations while maintaining alignment with actual camera poses.
Flow-Based Affordance Grounding: Proposes a generative flow-based model that predicts affordance distributions rather than single-mode predictions. This captures the inherent multi-modality of functional interactions and grounds affordances on the complete reconstructed shape, not just visible surfaces.
Affordance-Driven Active View Selection: Develops a strategy where predicted affordances guide active viewpoint selection. This creates a closed loop where functional understanding improves geometric reconstruction, which in turn refines affordance prediction, achieving significant gains with minimal additional views.

4. Experimental Results

The method was evaluated on the Toky4K (reconstruction) and Affogato (affordance grounding) benchmarks.

3D Reconstruction (Toky4K):
- Achieved 32.67 IoU, outperforming the state-of-the-art TRELLIS (19.49 IoU) by 67.6% and the depth-based MCC (21.11 IoU) by 54.8%.
- Demonstrated superior geometric accuracy and appearance quality (PSNR, LPIPS) compared to RGB-only generation and discriminative depth methods.
Complete 3D Affordance Grounding (Affogato):
- Achieved 19.1 aIoU (average Intersection over Union), a 40.4% improvement over the previous best (Espresso-3D at 13.6).
- Showed that generative modeling of distributions yields more precise spatial localization than discriminative approaches, even without task-specific encoder fine-tuning.
Partial 3D Affordance Grounding:
- When starting from a single RGB-D view, Affostruction achieved 9.26 aIoU on the complete object geometry.
- This is nearly 2x better than the best two-stage pipeline (MCC + Espresso-3D at 4.74), proving that joint generative reconstruction and grounding is superior to sequential pipelines.
Active View Selection:
- With just one additional view, the affordance-driven selection strategy achieved 2.0x faster improvement in aIoU compared to sequential baselines.
- It reached high performance (9.2 aIoU) with only 2 views, whereas sequential sampling required many more views to converge.
Efficiency:
- Total runtime is 7.20 seconds (4.74s for reconstruction + 2.46s for affordance), significantly faster than two-stage pipelines (e.g., MCC + Espresso-3D takes ~35s).
- Uses only 5 sampling steps due to the efficiency of the sparse voxel fusion, compared to 25 steps in standard diffusion models.

5. Significance

Bridging Perception and Action: Affostruction moves beyond "what is visible" to "what is functionally possible," enabling robots to reason about occluded parts of objects essential for manipulation.
Generative Paradigm for Robotics: It demonstrates that generative flow-based models, when combined with multi-view depth conditioning, can outperform discriminative methods in both geometric reconstruction and functional understanding.
Active Perception: The work validates the importance of active view selection driven by functional goals. By prioritizing views that reveal functional regions, robots can achieve high-fidelity understanding with minimal data acquisition, a critical capability for real-world robotic deployment where time and energy are constrained.
Handling Ambiguity: By modeling affordances as distributions, the framework naturally handles the uncertainty inherent in human-object interactions, providing a more robust foundation for robotic planning.