Why MLLMs Struggle to Determine Object Orientations

1. Problem Statement

Multimodal Large Language Models (MLLMs) consistently demonstrate poor performance in tasks requiring the identification of 2D object orientations (e.g., determining if an object is rotated 45°, 90°, or 180°). Previous literature (e.g., Tong et al., Nichols et al.) hypothesized that this failure stems from the visual encoder (such as CLIP, SigLIP, or ViT). The prevailing theory suggests that because these encoders are pre-trained for image-text semantic alignment rather than geometric reasoning, they fail to embed orientation information into their feature representations. Consequently, the MLLM has no data from which to recover the orientation.

2. Methodology

The authors designed a controlled empirical protocol to test the hypothesis that visual encoders lack orientation information. Instead of relying on the MLLM's language generation (which is known to fail), they isolated the visual encoder embeddings to see if orientation could be mathematically recovered from them.

Experimental Setup:

• Models Tested: Four open-source MLLMs using three distinct visual encoders:
  • LLaVA-OneVision: Uses SigLIP.
  • Qwen2.5-VL-7B-Instruct: Uses a native ViT.
  • LLaVA-1.5 & 1.6: Use CLIP.
• Image Datasets:
  • Whole Images: Rotated natural images (ImageNet) and in-place rotated objects (SI-Score dataset).
  • Blended Images: For single-image input models (LLaVA 1.5/1.6), the authors created synthetic images by superimposing circular foreground patches (e.g., a dog, lizard, train) onto natural backgrounds. This allowed for precise control of foreground rotation without boundary artifacts.
• Probing Mechanism:
  • The authors extracted high-dimensional embedding vectors from the visual encoders.
  • They trained Linear Ridge Regressors to predict the rotation angle (specifically the sine and cosine of the angle) directly from these embeddings.
  • The goal was to determine if a simple linear model could recover the ground-truth orientation from the encoder's output.

Feature Substitution Analysis:
To understand how the information is stored, the authors performed feature substitution experiments. They iteratively replaced features in a target embedding (with a specific rotation) with features from an anchor embedding (a different rotation) to see how many features needed to be swapped to "fool" the linear predictor into changing its prediction.

3. Key Contributions

1. Rejection of the Visual Encoder Hypothesis: The paper provides empirical evidence contradicting the widely held belief that visual encoders fail to encode orientation.
2. High-Fidelity Orientation Recovery: Demonstrated that linear regressors can predict object and image orientations from SigLIP, ViT, and CLIP embeddings with high accuracy (Mean Absolute Error < 3°).
3. Characterization of Error Distribution: Showed that prediction errors are approximately Gaussian (normal) and random, indicating the signal is robust and not merely noise.
4. Discovery of Diffuse Encoding: Revealed that orientation information is not localized to a few specific neurons but is diffusely distributed across tens of thousands of features in the embedding vector.
5. Background Sensitivity: Identified that foreground orientation estimation is highly sensitive to the canonical orientation of the background; rotating the background significantly degrades the ability to estimate the foreground's rotation.

4. Key Results

A. Linear Regression Performance
The linear regressors successfully predicted orientations with high precision across all tested models and datasets:

• LLaVA-OneVision (SigLIP) & Qwen2.5-VL (ViT): Achieved Mean Absolute Errors (MAE) generally < 1.5° for whole images and in-place rotated objects.
• LLaVA 1.5 & 1.6 (CLIP): Achieved MAE < 3° for blended foreground patches.
• Statistical Validity: Kolmogorov-Smirnov (K-S) tests confirmed that the residuals (errors) were normally distributed, suggesting the linear models captured the underlying signal effectively.

B. Feature Substitution Findings

• Diffuseness: To successfully change the predicted orientation from an anchor (e.g., 0°) to a target (e.g., 9°), the authors had to replace hundreds of thousands of features (e.g., >540,000 for LLaVA-OneVision, >128,000 for Qwen).
• Implication: Orientation is not encoded in a "sparse" or "localized" manner (like a specific "rotation neuron"). Instead, it is a global property distributed across the entire embedding space.

C. MLLM Failure Analysis
Despite the presence of accurate orientation signals in the embeddings, the MLLMs (the language model components) failed to utilize them.

• When prompted to answer orientation questions, MLLMs frequently guessed common angles (e.g., 90°) or claimed the task was impossible.
• The authors conclude that the failure lies not in the encoding of orientation, but in the decoding/exploitation of this diffuse information by the language model.

D. Background Dependency
Experiments with synthetic backgrounds showed that if the background is rotated while the foreground is static, the model's ability to estimate the foreground's orientation collapses (MAE > 80°). This suggests the encoder relies heavily on the background's canonical orientation as a reference frame.

5. Significance and Conclusion

This paper fundamentally shifts the understanding of MLLM limitations regarding spatial reasoning:

• The Bottleneck is Not the Encoder: The visual encoders (CLIP, SigLIP, ViT) are capable of encoding precise geometric orientation data. The "blindness" of MLLMs to orientation is not due to a lack of visual data.
• The Bottleneck is the Integration: The failure occurs in the multimodal integration layer. The language model struggles to interpret the diffuse, high-dimensional orientation signals embedded in the visual features. The information is "there," but it is too scattered for the current architecture to easily attend to and reason about.
• Future Directions: The findings suggest that improving MLLM spatial reasoning may require architectural changes to better aggregate diffuse features or specific training objectives that force the language model to learn how to "read" these distributed orientation signals, rather than simply retraining the visual encoder.

In summary, the paper proves that MLLMs have the visual data to solve orientation tasks but lack the reasoning mechanism to extract it, challenging the community to look beyond visual encoders when diagnosing spatial intelligence failures.