SceneFunRI: Reasoning the Invisible for Task-Driven Functional Object Localization

Technical Summary: SceneFunRI – Reasoning the Invisible for Task-Driven Functional Object Localization

Problem Definition
Current Vision-Language Models (VLMs) struggle with spatial reasoning in real-world scenarios where target objects are entirely invisible. While humans can infer the locations of occluded functional objects using context and commonsense knowledge, existing benchmarks and models largely rely on visible cues or partial occlusion (e.g., shadows or reflections). The paper identifies a gap in evaluating the ability to reason about "invisible" objects—targets that are completely outside the visual field and lack indirect visual evidence, requiring inference solely from task instructions, scene layout, and commonsense priors.

Methodology
The authors introduce SceneFunRI, a benchmark specifically designed to evaluate "Reasoning the Invisible" (RI).

• Dataset Construction: Built upon the SceneFun3D dataset, the benchmark utilizes a semi-automatic pipeline to generate 855 task-driven instances. The process involves:
  1. Frame Selection: Identifying frames where a functional object exists within the camera's view but is fully occluded ($V_o = 0\%$) in the 2D image.
  2. Ground Truth Generation: Projecting 3D point cloud annotations of the hidden objects onto the 2D image plane to create ground-truth bounding boxes.
  3. Quality Control: Human verification ensures tasks have sufficient contextual cues (e.g., the presence of a bed to infer a nightstand's location) and unambiguous instructions, filtering the initial 1,232 candidates down to 855 high-quality instances.
• Evaluation Metrics: Standard Intersection over Union (IoU) is deemed too strict for invisible targets. The authors propose:
  • Containment Accuracy (CAcc@$\theta$): Measures if the ground-truth box is contained within the predicted region, penalizing overly large predictions.
  • Dist: A normalized corner-distance metric to evaluate localization proximity even when boxes do not overlap.
• Prompting Strategies: The study analyzes three prompting categories:
  1. Strong Instruction Prompting: Explicitly instructing models to ignore visible objects and focus on invisible regions or apply commonsense rules.
  2. Reasoning-based Prompting: Utilizing Chain-of-Thought (CoT) and CoT with Self-Reflection (CoT-SR) to encourage step-by-step inference.
  3. Spatial Process of Elimination (SPoE): An iterative framework inspired by human decision-making. The model progressively partitions the image into regions, eliminates low-probability candidates, and narrows the search space until the target location is localized.

Key Results

• Model Performance: Even the strongest baseline, Gemini 3 Flash, achieves a low CAcc@75 of 15.20 and a Dist of 28.65, significantly trailing the human baseline (CAcc@75 of 66.43). Open-source models (Qwen, InternVL, Gemma) and specialized spatial reasoning models generally perform worse, indicating that current VLMs lack robust capabilities for invisible reasoning.
• Prompting Analysis:
  • Strong Instructions: The "Hidden" prompt (ignoring visible objects) yielded stable improvements, suggesting latent reasoning capabilities. However, adding explicit commonsense knowledge ("Common" prompt) often degraded performance, highlighting a disconnect between language understanding and spatial grounding.
  • Reasoning: CoT and CoT-SR improved performance only for larger models (Qwen 3.5 27B/122B) that lack automatic structured reasoning, whereas models like Gemini 3 Flash (which inherently generate structured reasoning) saw no gain or slight degradation.
  • SPoE: The SPoE strategy effectively reduced the Dist metric as iterations increased, demonstrating that models are better at excluding low-probability visible regions than directly inferring invisible locations. However, models struggled to determine an optimal stopping point, often continuing to eliminate regions even after the search space was sufficiently constrained.
• Qualitative Findings: Models frequently misidentified objects with similar visual appearances or semantic associations (e.g., confusing a power switch for a socket) and failed to infer fine-grained details like the position of occluded door handles, even when identifying the general task area.

Significance and Claims
The paper claims that SceneFunRI exposes a fundamental limitation in current VLMs: the inability to integrate task intent, commonsense priors, and spatial grounding to reason under conditions of complete visual uncertainty.

• Benchmark Contribution: It extends spatial reasoning research from visible, explicit relations to invisible, implicit inference, providing a necessary testbed for embodied agents and real-world decision-making.
• Methodological Insight: The authors argue that directly inferring hidden regions is a substantial challenge. Instead, the paper suggests that a more viable approach for current models may be analyzing observable regions to exclude low-probability candidates (as demonstrated by SPoE), rather than attempting direct generation of invisible locations.
• Future Direction: The results motivate the development of models that can better integrate uncertainty-aware search and tighter coupling between semantic intent and spatial grounding, as current models remain unstable in reasoning about invisible regions.