Which Reconstruction Model Should a Robot Use? Routing Image-to-3D Models for Cost-Aware Robotic Manipulation

1. Problem Statement

Robotic manipulation tasks require 3D mesh reconstructions of varying quality depending on the specific sub-task:

• Dexterous manipulation requires fine-grained surface details.
• Collision-free planning can tolerate coarser representations.

Current Image-to-3D models (e.g., TripoSR, Hunyuan3D, TRELLIS) offer different trade-offs between inference speed, memory usage, and reconstruction fidelity. Their performance is highly viewpoint-dependent (a frontal view may yield poor results for some models, while a corner view yields excellent results). Additionally, view-invariant methods (e.g., structured light scanning) exist but incur different costs (time, hardware).

The Challenge:

• Querying all available models for every input is computationally prohibitive.
• Existing routing methods (mostly from LLM literature) are ill-suited for robotics because:
  1. They typically assume a single scalar cost trade-off, whereas robotics requires multi-dimensional cost constraints (latency, memory, monetary cost).
  2. They often fail to distinguish between viewpoint-dependent models (which change based on the input image) and view-invariant methods (which depend on hardware configuration but not the specific input view).
  3. Adding new view-invariant configurations usually requires retraining the entire routing network.

2. Methodology: SCOUT Framework

The authors propose SCOUT (Score-Conditioned Optimal Utility Targeting), a routing framework designed to select the optimal reconstruction model for a given input image under arbitrary cost constraints.

Core Innovation: Score Decoupling

SCOUT decouples the reconstruction score $s(x, m)$ into two distinct components:

1. Relative Performance ($\hat{p}$): A learned probability distribution capturing how viewpoint-dependent models perform relative to each other for a specific input image.
2. Image Difficulty ($\hat{z}$): A scalar partition function estimate capturing the overall difficulty of the input image (e.g., a "hard" frontal view vs. an "easy" corner view).

The estimated score for a model $m_j$ is reconstructed as:
$$\hat{s}(x, m_j) = \begin{cases} T \cdot \ln(\hat{p}_j \cdot \hat{z}) & \text{if } m_j \text{ is viewpoint-dependent} \\ s_{inv}(m_j) & \text{if } m_j \text{ is view-invariant} \end{cases}$$

Key Technical Components

1. Preprocessing & Smoothing:

   • Reconstruction scores are grouped by semantic object categories to compute tag-averaged scores.
   • A smoothing parameter $\beta$ blends raw scores with category averages to reduce noise in the training targets for relative performance.

2. Learning Relative Performance ($\hat{p}$):

   • Instead of regressing absolute scores (which confuses image difficulty with model capability), SCOUT trains a neural network to predict the softmax distribution of the smoothed scores using KL Divergence.
   • This forces the network to learn which model is best for a specific view, independent of how hard the view is.

3. Estimating Image Difficulty ($\hat{z}$):

   • To recover absolute scores for comparison with view-invariant methods, SCOUT estimates a partition function $z$.
   • It uses a proxy derived from the unsmoothed scores and the predicted probabilities: $\tilde{z}_j = e^{s_j/T} / \hat{p}_j$.
   • Optimal Weighting: Since there are $k_1$ proxies for $z$ (one per viewpoint-dependent model), SCOUT derives a theoretically optimal weighted average to combine them, minimizing variance based on the noise characteristics of each model's predictions.

4. Utility Optimization:

   • At inference time, the system selects the model $m^*$ that maximizes:
     $$m^* = \arg\max_{m \in M} (\hat{s}(x, m) - c(m))$$
   • Here, $c(m)$ is an arbitrary cost coefficient vector (e.g., $[latency, memory]$).
   • Because the network only predicts $\hat{p}$ over viewpoint-dependent models, view-invariant methods can be added, removed, or reconfigured at inference time without retraining.

3. Key Contributions

1. First Formulation of Model Routing for 3D Reconstruction: Addresses the unique constraints of robotics, including viewpoint dependency and multi-dimensional cost vectors.
2. Theoretical Partition Function Proxy: Introduces a method to recover absolute scores from relative distributions with provably optimal weighting, enabling comparison between dependent and independent methods.
3. Decoupling Procedure: Separates image difficulty from relative model performance, allowing the system to scale to new view-invariant pipelines without retraining.
4. Cost-Aware Flexibility: Supports arbitrary cost constraints at inference time, unlike previous LLM routing methods that require retraining for new cost trade-offs.

4. Experimental Results

The framework was evaluated on Google Scanned Objects (GSO), BigBIRD, and YCB datasets using metrics like Density-aware Chamfer Distance (DCD), IoU, and Eval3D geometric metrics.

• Routing Performance:

  • SCOUT consistently outperformed baselines adapted from LLM literature (ZOOTER, RouterDC, MLP, kNN, Linear Regression) across all cost subspaces.
  • It achieved the lowest regret (difference between selected model utility and optimal oracle utility) in challenging cost regions.
  • Decoupling Impact: Removing the decoupling mechanism (forcing a joint softmax over all models) significantly degraded performance as the number of view-invariant methods increased.

• Robotics Validation:

  • Grasp Proposals: SCOUT-selected models resulted in lower grasp collision rates and higher mesh-IoU compared to using a single fixed model.
  • Dexterous Manipulation: In simulation, SCOUT improved success rates for complex grasps that rely on surface geometry.
  • Real-World Deployment: Tested on a Franka Panda robot for pick-and-place tasks. SCOUT successfully routed difficult views to high-fidelity models (e.g., Hunyuan3D) and easy views to faster models (e.g., TRELLIS), achieving a 4.6/5 success rate compared to lower rates for fixed-model baselines.

5. Significance

This work bridges the gap between generative 3D vision and practical robotic manipulation. By introducing a routing framework that is cost-aware, scalable, and viewpoint-sensitive, SCOUT enables robots to dynamically select the most appropriate perception pipeline based on real-time constraints (time, memory) and task requirements (precision vs. speed).

The ability to add new reconstruction methods (especially hardware-based, view-invariant ones) without retraining makes this framework highly practical for real-world deployment where sensor configurations may change. It demonstrates that intelligent model selection can yield tangible improvements in downstream robotic tasks, moving beyond the "one-model-fits-all" paradigm currently prevalent in robotic perception.