Scalable Object Relation Encoding for Better 3D Spatial Reasoning in Large Language Models

1. Problem Statement

The paper addresses the challenge of enabling Large Language Models (LLMs) to perform 3D spatial reasoning (locating objects based on spatial relations) in 3D Vision-Language (VL) tasks. Existing approaches face three primary limitations:

• Data Scarcity: There is a lack of 3D scene-text paired data, making it difficult to train models with strong reasoning capabilities from scratch.
• Ineffective Absolute Position Encoding: Methods that inject absolute 3D coordinates directly into object features often fail because absolute origins lack physical meaning in 3D space. Furthermore, "premature feature fusion" (mixing coordinates with object features before the LLM) prevents the model from explicitly learning pairwise relationships.
• Scalability Issues with Explicit Relations: Methods that explicitly encode all pairwise spatial relations as input tokens suffer from quadratic complexity ($O(n^2)$). With scenes containing hundreds of objects, this exceeds LLM input limits. Pruning strategies (e.g., keeping only nearest neighbors) risk discarding critical spatial relations, leading to reasoning errors.
• Interference with Language RoPE: Combining 3D positional embeddings with existing Language Rotary Positional Embeddings (RoPE) in LLMs causes interference, degrading both spatial and linguistic performance.

2. Methodology

The authors propose QuatRoPE (Quaternion Rotary Positional Embedding) and IGRE (Isolated Gated RoPE Extension) to solve these issues.

A. QuatRoPE: Scalable Pairwise Relation Encoding

Instead of encoding absolute positions or explicit relation tokens, QuatRoPE encodes absolute 3D coordinates into object-related tokens and leverages the Transformer's attention mechanism to derive relative positions dynamically.

• Holistic Vector Encoding: Unlike standard RoPE which treats axes independently, QuatRoPE treats the 3D coordinate $(x, y, z)$ as a holistic vector. It applies quaternion rotations to the Query ($\vec{q}$) and Key ($\vec{k}$) vectors in the attention layer based on the object's 3D position.
• Mathematical Mechanism:
  • The rotation function $f(\vec{x}, \vec{p})$ rotates a vector $\vec{x}$ by the position $\vec{p}$ using quaternions.
  • The core insight is that the dot product (attention score) between two rotated vectors depends solely on their relative position ($\vec{m} - \vec{n}$), not their absolute positions.
  • Formula: $\langle f(\vec{q}, \vec{m}), f(\vec{k}, \vec{n}) \rangle = g(\vec{q}, \vec{k}, \vec{m}-\vec{n})$.
• Benefits:
  • Linear Scalability: Input length remains $O(n)$ (one token per object) while implicitly capturing all $O(n^2)$ pairwise relations.
  • Geometric Integrity: By encoding coordinates as unified vectors, it prevents "false proximity" errors where objects are close on one axis but far in 3D space (a flaw in axis-independent methods).
  • Maxim of Relation: It naturally increases attention scores for spatially proximate objects, aligning with human cognitive biases for implicit references (e.g., "the window to the left" implies the closest one).

B. IGRE: Isolated Gated RoPE Extension

To integrate QuatRoPE with pre-trained LLMs that already use Language RoPE without causing interference:

• Dimension Isolation: QuatRoPE is applied to a specific set of dimensions in the Query/Key vectors, while Language RoPE operates on the original dimensions.
• Gating Mechanism:
  • For object tokens, the QuatRoPE dimensions are populated with rotated values.
  • For non-object tokens (e.g., prompts, questions), these dimensions are zero-padded.
• Effect: This ensures that the spatial rotation only affects attention scores when two object tokens interact. Non-object tokens do not receive spatial rotation, preventing the model from falsely associating text prompts with the coordinate origin $(0,0,0)$. This preserves the LLM's original linguistic capabilities while adding spatial reasoning.

C. Attribute-free Spatial Reasoning (ASR) Benchmark

To rigorously evaluate spatial reasoning without bias from object attributes (color, category), the authors constructed a new benchmark:

• Construction: Derived from ScanQA, filtering for questions where the target object is identified only by its spatial relation to anchors, removing all attribute descriptions.
• Format: Converted to a 3D Visual Grounding (VG) multiple-choice format to eliminate language generation biases.

3. Key Contributions

1. QuatRoPE: A novel positional embedding method that achieves $O(n)$ input length while explicitly modeling $O(n^2)$ pairwise spatial relations via quaternion rotations in attention layers.
2. IGRE: A mechanism to seamlessly combine QuatRoPE with Language RoPE, isolating spatial and linguistic dimensions to prevent interference and preserve pre-trained capabilities.
3. ASR Benchmark: A diagnostic benchmark designed to exclusively evaluate 3D spatial reasoning by removing confounding attribute cues.
4. State-of-the-Art Performance: Demonstrated consistent and significant improvements over strong baselines (Chat-Scene, 3DGraphLLM) across multiple 3D VL tasks.

4. Experimental Results

The authors evaluated their approach on ScanRefer, Multi3DRef, SQA3D, and the new ASR benchmark.

• Comparative Performance:
  • On ScanRefer (3D Visual Grounding), QuatRoPE + 3DGraphLLM-7B achieved 58.30% Acc@0.25, outperforming the baseline (57.0%) and other SOTA models.
  • On SQA3D (Situated QA), similar gains were observed, with the 7B model reaching 55.2% EM@1.
  • The method showed consistent gains across different model sizes (1B and 7B) and architectures.
• ASR Benchmark Results:
  • QuatRoPE provided significant gains in pure spatial reasoning tasks (e.g., +19.48% gain on Chat-Scene 1B), proving the model learns spatial relations rather than relying on object attributes.
• Ablation Studies:
  • IGRE vs. Additive: IGRE outperformed simple additive RoPE combinations, confirming the necessity of dimension isolation.
  • Holistic vs. Raw Coordinates: Raw coordinate injection performed poorly (especially on 3DGraphLLM), while QuatRoPE (holistic) significantly outperformed M-RoPE (multi-axis RoPE), validating the importance of holistic vector encoding.
  • False Proximity: QuatRoPE showed increasing advantages as the "false proximity" severity (objects close on one axis but far in 3D) increased.

5. Significance

This paper provides a scalable and mathematically rigorous solution for integrating 3D spatial reasoning into LLMs.

• Scalability: It solves the quadratic complexity bottleneck, making it feasible to process complex 3D scenes with hundreds of objects without pruning critical relations.
• Generalizability: The IGRE mechanism allows leveraging powerful pre-trained LLMs without retraining from scratch, a crucial advantage given the scarcity of 3D data.
• Cognitive Alignment: By mimicking human spatial attention (Maxim of Relation), the model improves its ability to handle implicit references in natural language.
• Evaluation Standard: The introduction of the ASR benchmark sets a new standard for evaluating true spatial reasoning capabilities, separating them from semantic recognition.

In summary, QuatRoPE transforms the way 3D geometry is represented in LLMs, moving from explicit, expensive relation tokens to an efficient, attention-driven relative positioning system that significantly enhances embodied AI capabilities.