Technical Summary: FUSE – Active Functional Affordance Grounding

Problem Definition: Active Functional Affordance Grounding

The paper addresses a critical gap in embodied AI: the ability to identify and interact with objects based on their function rather than their pre-defined identity. Existing methods typically operate from fixed viewpoints or assume sufficient visual evidence is already available. However, functional queries (e.g., "cut a piece of rope" or "scoop food") often generate multiple plausible hypotheses where the necessary discriminative evidence is occluded, incomplete, or unavailable from the initial view.

The authors define Active Functional Affordance Grounding as a sequential perception task where an agent must:

1. Infer candidate object hypotheses from a functional language query (without a predefined target class).
2. Actively select sensing actions (viewpoints) to resolve semantic and geometric ambiguities.
3. Spatially ground the object that best satisfies the functional intent before physical execution.

This differs from passive grounding (which may commit prematurely), active object search (which seeks a predefined category), and next-best-view planning (which assumes a known target).

Methodology: The FUSE Framework

The authors propose FUSE (Functional Understanding through Semantic–Geometric Exploration), an adaptive framework designed to acquire informative observations efficiently. FUSE operates by minimizing functional uncertainty, defined as the unresolved semantic and geometric evidence required to ground a functional target.

Core Components

1. Functional Uncertainty Representation:

   • Semantic Uncertainty ($U_{sem}$): Derived from SAM3, which generates pixel-level masks and scores for candidate object hypotheses. Uncertainty is estimated as $1 - \text{score}$.
   • Geometric Uncertainty ($U_{geo}$): Derived from SSNR (Structural Signal-to-Noise Ratio), calculated by comparing gradients between the observed image and a rendered image from a 3D scene representation. Low SSNR indicates incomplete or inaccurate structural modeling.
   • Evidence Map: These components are fused into a functional evidence acquisition map ($E_t$) that balances candidate relevance with geometric under-observation.

2. Spatial Evidence Representation:

   • The agent maintains a persistent, incrementally refined 3D Gaussian Splatting (3DGS) model. This allows the agent to reason about visibility, occlusion, and geometric structure across viewpoints, rather than treating observations as a disconnected sequence.

3. Adaptive Planning Strategy:
   FUSE employs a confidence-gated mechanism to switch between two planning modes:

   • Amortized Planner: A learned policy (CLIP backbone + MLP) that predicts the value of sensing actions in a single forward pass. It is fast but may be ambiguous in unfamiliar states.
   • Explicit Exploration: A deliberative process that updates the 3DGS representation, renders candidate viewpoints, and computes the full semantic-geometric evidence map to select the optimal action.
   • Switching Logic: The system calculates the entropy of the amortized planner's action distribution. If entropy is low (high confidence), the amortized action is selected. If entropy is high (uncertainty), the system triggers explicit exploration to refine the scene representation and re-evaluate.

4. Termination:
   Exploration stops when the fused target score (SAM3 score + depth-proximity score) fails to improve for $K$ consecutive steps or the sensing budget is exhausted.

Key Contributions

The paper outlines four primary contributions:

1. Task Formulation: The definition of Active Functional Affordance Grounding as a new sequential perception task requiring target discovery and spatial grounding under partial observability.
2. Benchmark: The introduction of a Habitat-based benchmark with 100 randomized scene instances, standardized stopping criteria, and evaluation tracks comparing static, canonical, random, VLM-based, and adaptive methods.
3. Algorithm: The development of FUSE, a framework that adaptively alternates between fast amortized planning and explicit semantic-geometric search to balance computational cost and grounding reliability.
4. Empirical Validation: Controlled evaluations demonstrating that task-conditioned semantic-geometric evidence acquisition improves active grounding across diverse knowledge sources.

Experimental Results

Experiments were conducted on the proposed Habitat benchmark using CRAFT-E as the primary affordance knowledge source, with comparisons against other LLMs and VLMs.

• Performance vs. Baselines:
  • Passive/Static: Achieved only 42% success rate.
  • Random Active: Improved to 56% success but required significantly more steps (12.20) and lacked task direction.
  • VLM Active: Reached 65% success using an oracle label and foundation model guidance but still lagged behind FUSE in final grounding quality (SAM3 score).
  • FUSE: Achieved the highest non-oracle Success Rate (72%) and Average IoU (70.91%).
• Efficiency:
  • FUSE reduced computational cost by 1.33× relative to fully explicit exploration while maintaining comparable performance (72% vs. 70% success).
  • It invoked explicit exploration for only 33.41% of decisions, relying on the amortized planner for routine steps.
• Robustness:
  • FUSE remained effective across various knowledge sources (CRAFT, GPT, Llama, Qwen, Sonnet, Gemini), indicating its strategy is decoupled from the specific upstream reasoning mechanism.
  • The "Oracle-Label" variant achieved 77% success, suggesting that while FUSE is robust, the quality of the initial functional hypothesis generation remains a limiting factor.
• Ablation Studies:
  • Semantic vs. Geometric: Semantic-only grounding (53%) was insufficient. Geometric-only (65%) was better but still inferior to the fused approach (72%).
  • Granularity: Pixel-level semantic evidence (SAM3) was crucial; patch-level (CLIP) dropped success to 34%.
  • Termination: Incorporating depth-proximity into the stopping criterion improved success by ~5% over SAM3 alone.

Significance and Claims

The authors claim that FUSE represents a significant step forward in embodied perception by addressing the "what to look for" and "where to look" problems simultaneously without predefined targets.

• Beyond Passive Grounding: The paper argues that functional understanding requires active evidence acquisition; relying on fixed views or single observations leads to premature commitment and failure in occluded scenarios.
• Efficiency in Active Perception: By adaptively switching between learned (amortized) and explicit planning, FUSE demonstrates that agents can achieve high-fidelity grounding without the prohibitive computational cost of continuous explicit scene reconstruction.
• Generalizability: The framework is shown to be robust across different functional reasoning backends, suggesting that the mechanism of adaptive evidence acquisition is a generalizable solution for functional grounding tasks.

The paper concludes that while current methods have improved functional reasoning from available observations, FUSE explicitly solves the preceding perception problem: determining what additional observations are necessary to reliably ground a functional intent. Future work is proposed to validate FUSE on real robotic platforms and extend it to dynamic environments and long-horizon tasks.