Technical Summary: OrientSAM

Problem Statement

Multimodal Large Language Models (MLLMs) exhibit significant limitations in spatial reasoning tasks that require perspective transformation. While current models perform well on tasks interpretable directly from the camera view, they struggle when the correct answer depends on the local viewpoint of a reference object (allocentric reasoning) rather than the image plane (egocentric reasoning).

The authors identify a specific failure mode termed the "camera-centric shortcut." In scenarios where the spatial relationship defined by the reference object's local frame conflicts with the horizontal arrangement in the image plane, MLLMs consistently default to the camera-centric cue. The paper demonstrates that this shortcut is not merely a general spatial reasoning deficit but is tightly coupled with the orientation of the reference object. Specifically, models fail to perform the necessary reference-frame transformation when the reference object's facing direction diverges from the camera's viewpoint, indicating a lack of explicit orientation modeling in their representations.

Methodology: OrientSAM

To address this, the authors propose OrientSAM (Orientation-aware Spatial Alignment), a framework designed to inject explicit orientation information into multimodal representations and align it with downstream spatial reasoning. The methodology consists of three core components:

1. Multi-Expert Spatial Data Construction

Since large-scale orientation-aware spatial supervision is scarce, the authors build a pipeline to automatically generate training data from large-scale 2D images. This pipeline integrates:

• Region Grounding: Using models like GroundingDINO and SAM to extract object bounding boxes and segmentation masks.
• Monocular Geometry Estimation: Estimating depth maps and camera intrinsics to derive instance-level depth.
• Orientation Prediction: Using an orientation estimation model to predict semantic facing directions (polar, azimuth, and rotation angles) for each object.
• Curriculum Organization: The data is filtered by confidence and organized into two stages: an orientation perception stage (learning stable orientation primitives) and a relation reasoning stage (learning reference-frame transformation).

2. Orientation-Aware Representation

OrientSAM explicitly encodes object orientation into the multimodal input space, moving beyond treating orientation as a mere textual hint.

• Fourier-Based Angle Encoding: To handle the periodic nature of angles (where $0^\circ$ and $360^\circ$ are identical), the authors employ Fourier feature encoding ($\gamma(\phi)$) rather than direct scalar regression.
• Token Injection: The encoded orientation is injected into both visual and textual representations:
  • Visual: The orientation encoding is projected and added to the region-level RGB features ($\tilde{r}_i = r_i + W_o\gamma(\phi_i)$).
  • Textual: The object token is augmented with the same encoding ($\tilde{e}_i = \text{Embed}(t_i) + W_t\gamma(\phi_i)$).
• Structured Alignment: The model uses disentangled tokens for appearance ($\langle r_{gbi} \rangle$), depth ($\langle depth_i \rangle$), and orientation ($\langle orient_i \rangle$). This allows the language model to dynamically attend to the specific geometric cues required for reasoning, rather than collapsing them into a single vector.

3. Curriculum Learning Strategy

The training process follows a progressive curriculum to stabilize learning:

• Stage 1 (Orientation Perception): The model is trained to predict the azimuth angle of individual objects, establishing a robust mapping between visual appearance and semantic facing direction.
• Stage 2 (Orientation-Aware Reasoning): The model is trained on full perspective-taking tasks, predicting spatial relations under the reference object's coordinate system using the structured RGB-depth-orientation tokens. This prevents the model from converging to camera-centric shortcuts by forcing it to rely on the learned orientation priors.

Key Contributions

1. Diagnostic Framework: The authors introduce a unified framework to disentangle reference-centric answers (ground truth under the object's view) from camera-centric answers (image-plane relations). Analysis reveals that current MLLMs exhibit a strong camera-centric shortcut, particularly when these two answers conflict, and that this shortcut strength varies with the reference object's orientation.
2. OrientSAM Framework: A novel alignment framework that explicitly incorporates orientation priors into multimodal representations via Fourier-based encoding and structured tokens, addressing the bottleneck of reference-frame transformation.
3. Data Construction Pipeline: A scalable, multi-expert pipeline that generates orientation-aware spatial supervision from large-scale 2D images, enabling the training of models on perspective-taking tasks without manual 3D annotation.
4. Empirical Validation: Extensive experiments demonstrate that explicit orientation modeling is critical for robust allocentric reasoning.

Experimental Results

OrientSAM was evaluated on three benchmarks: Spatial-MM, ViewSpatial, and 3DSRBench.

• Performance Gains: OrientSAM consistently outperforms strong baselines (including LLaVA-v1.5, Qwen2.5-VL, Qwen3-VL, SpatialRGPT, and APC-Vis).
  • On Spatial-MM, it achieves 87.31% accuracy on non-camera-view tasks, significantly surpassing the best baseline (APC-Vis at 86.57%).
  • On ViewSpatial, it achieves state-of-the-art results on three of four subtasks, with a notable margin on person-view relative direction (85.04% vs. 72.57% for APC-Vis).
  • On 3DSRBench, it leads in Orientation & Direction (81.83%) and Multi-object Relation (77.80%) tasks.
• Ablation Studies:
  • Orientation Injection: Removing the orientation branch drops performance significantly (e.g., from 87.31% to 76.29% on Spatial-MM non-camera-view), confirming that data fine-tuning alone is insufficient.
  • Curriculum Learning: Removing the curriculum strategy reduces performance, indicating that progressive training from perception to reasoning is essential for stability.
  • Encoding Strategy: Fourier-based encoding (specifically $N=2$) outperforms direct scalar encoding, validating the need to model the periodic nature of angles.
• Shortcut Analysis: The study confirms that OrientSAM effectively mitigates the camera-centric shortcut. On conflict samples, the model's agreement with the reference-centric ground truth increases while its reliance on camera-centric cues decreases, particularly in front-facing and back-facing regimes where previous models failed.

Significance and Claims

The paper claims that the primary bottleneck for perspective-aware spatial reasoning in MLLMs is not a lack of general visual perception or generic geometric cues, but the absence of explicit orientation modeling.

The authors argue that:

1. Camera-Centric Bias is Systematic: Current models rely on image-plane heuristics rather than true perspective-taking, a behavior that is exacerbated when the reference object's orientation conflicts with the camera view.
2. Orientation is a Missing Inductive Bias: By explicitly injecting orientation priors into the representation space, models can learn to transform spatial relations into local reference frames, enabling robust allocentric reasoning.
3. Practical Implications: This approach is crucial for applications requiring embodied interaction, human-object collaboration, and scene understanding, where reasoning must occur from the viewpoint of an entity in the scene rather than the camera.

The authors remain modest regarding limitations, acknowledging that the framework currently relies on the quality of upstream expert models (grounding, depth, orientation) and primarily focuses on left/right spatial reasoning. They suggest future work should address richer spatial relations and reduce dependence on external annotations.