A3R: Agentic Affordance Reasoning via Cross-Dimensional Evidence in 3D Gaussian Scenes

1. Problem Statement

Affordance reasoning involves identifying specific regions in a 3D scene that support a given action (e.g., "grasp the handle of the kettle"). While recent advances in 3D Gaussian Splatting (3DGS) have enabled efficient, geometry-aware scene representations, existing methods for affordance reasoning suffer from a critical limitation: evidence insufficiency under static observation.

• The Core Issue: Most current approaches treat affordance reasoning as a "one-shot" prediction problem, assuming a single fixed observation contains sufficient information. However, in complex 3D scenes, multiple objects or parts often share similar local geometries, or the target affordance lies in a structurally ambiguous region.
• The Consequence: Static models fail to distinguish between functionally similar candidates because they lack the specific task-relevant evidence needed to resolve ambiguity. They often rely on static fusion of 2D semantics and 3D geometry, which can propagate noise or fail to address the specific uncertainty of the current query.

2. Methodology: A3R Framework

The authors propose A3R (Agentic Affordance Reasoning), a framework that reformulates affordance reasoning as a sequential cross-dimensional evidence acquisition process. Instead of predicting the answer immediately, an agent iteratively gathers complementary evidence to reduce uncertainty.

A. Problem Formulation (POMDP)

The task is modeled as a Partially Observable Markov Decision Process (POMDP):

• Latent State ($Y$): The true affordance assignment on the 3D Gaussians.
• Observation: The agent cannot see $Y$ directly but acquires evidence through specific operators.
• Decision State ($s_t$): A scene-grounded state comprising:
  1. Current Workspace: The subset of the 3D scene currently under focus.
  2. Affordance Belief ($h_t$): A probability map over Gaussians indicating the likelihood of being the target.
  3. History ($m_t$): The sequence of past actions and acquired evidence.

B. Agentic Policy (MLLM-Driven)

A Multi-Modal Large Language Model (MLLM) acts as the policy ($\pi_\theta$).

• Input: The policy receives a multimodal context including rendered visual views of the current workspace, the current belief map, the user query, and a textual summary of the history.
• Output: The policy selects the next evidence acquisition action from a predefined schema.

C. Cross-Dimensional Evidence Operators

The agent can dynamically select from five operators to acquire specific types of evidence:

1. 3D Instance Segmentation (Inst): Identifies the relevant object instance to suppress distractors (Geometric).
2. 3D Part Segmentation (Part): Predicts the specific part support for the affordance (Geometric).
3. 2D Semantic Segmentation with 3D Lifting (SAM2D): Renders 2D views, segments them using a foundation model (e.g., SAM), and lifts the mask back to 3D (Semantic).
4. 3D Zoom: Narrows the workspace to a local sub-scene to refine spatial granularity.
5. Terminate: Stops the process and outputs the final prediction based on the current belief threshold.

D. State Transition & Belief Update

• When an operator executes, its output is projected onto the 3D Gaussian support.
• Belief Update: Geometric/semantic operators (Part, SAM2D) directly update the affordance belief map ($h_t$). Workspace control operators (Inst, Zoom) refine the workspace but leave the belief unchanged until new evidence is projected.
• This mechanism allows the agent to "query on demand," acquiring 2D semantics only when geometry is ambiguous, or zooming in when the region is too coarse.

E. Optimization (GRPO)

To train the policy without step-by-step supervision, the authors use Group Relative Policy Optimization (GRPO):

• The policy generates a group of trajectories (rollouts).
• Rewards are calculated based on the final accuracy (IoU with Ground Truth) and a penalty for the number of steps taken.
• GRPO computes a group-relative advantage to optimize the policy, encouraging trajectories that achieve high accuracy with fewer steps.

3. Key Contributions

1. Problem Reformulation: Identified that fine-grained affordance reasoning failures are often due to insufficient evidence rather than weak prediction models, shifting the paradigm from static one-shot prediction to sequential evidence acquisition.
2. A3R Framework: Proposed an agentic framework that dynamically queries complementary 3D geometric and 2D semantic evidence using an MLLM-based policy, resolving ambiguity progressively.
3. GRPO-Based Learning: Developed a training strategy using Group Relative Policy Optimization that optimizes the entire trajectory of evidence acquisition based on final task rewards, eliminating the need for intermediate supervision.
4. Cross-Dimensional Synergy: Demonstrated that dynamically coordinating 2D and 3D tools (rather than static fusion) significantly improves performance in complex, ambiguous scenes.

4. Experimental Results

The method was evaluated on two 3DGS-based benchmarks: 3DAffordSplat (instance-level) and SeqAffordSplat (scene-level).

• Performance: A3R consistently outperformed state-of-the-art static baselines (e.g., AffordSplatNet, PointRefer, IAGNet).
  • On the Unseen setting of SeqAffordSplat, A3R (with RL) achieved 25.56 mIoU, significantly surpassing the best static baseline (9.01 mIoU).
  • On 3DAffordSplat, A3R achieved 26.27 mIoU in the Unseen setting compared to 17.31 mIoU for the best baseline.
• Efficiency: The agent learned to resolve ambiguity efficiently, averaging 2.60 steps per query, compared to fixed-sequence baselines that often took 3+ steps or failed.
• Ablation Studies:
  • Complementarity: Using only 3D geometry or only 2D semantics performed worse than the full model, proving the necessity of cross-dimensional evidence.
  • Dynamic vs. Static: The adaptive policy significantly outperformed "Static Fusion" and "Fixed Sequence" strategies, confirming that the order and selection of evidence matter.
  • Backbone: Larger MLLM backbones (Qwen3-VL-8B) yielded better accuracy and required fewer interaction steps.

5. Significance

This work represents a significant shift in 3D scene understanding:

• From Passive to Active: It moves beyond passive observation, treating reasoning as an active process of gathering information, similar to human perception.
• Robustness in Ambiguity: It solves a critical failure mode in current 3D AI: the inability to distinguish between similar-looking objects or parts when a single view is insufficient.
• Generalization: The ability to generalize to "Unseen" affordances suggests that the agent learns a reasoning strategy (how to gather evidence) rather than just memorizing visual patterns.
• Future Impact: This approach provides a blueprint for embodied AI agents that need to interact with complex, real-world 3D environments where perfect information is rarely available in a single glance.