Technical Summary: Distilling Neuro-Symbolic Programs into 3D Multi-modal LLMs (APEIRIA)

1. Problem Statement

Current approaches to 3D spatial reasoning face a fundamental trade-off between interpretability and flexibility:

• Neuro-Symbolic 3D (NS3D) Learners: These methods achieve interpretable, systematic reasoning by decomposing queries into compositional programs executed by concept-specific networks. However, they are constrained to closed-set vocabularies (unable to handle open-vocabulary concepts like "cozy chair") and require dense, step-by-step supervision that is often unavailable in real-world datasets.
• 3D Multi-modal Large Language Models (3D MLLMs): These models leverage the semantic flexibility of LLMs to handle complex, open-vocabulary natural language instructions. However, they operate as black boxes, lacking explicit reasoning structures or verification steps. When they fail, there is no trace to diagnose whether the error lies in object recognition, relation understanding, or logical composition.

The paper posits that the field is trapped between interpretable but limited NS3D methods and powerful but opaque MLLMs. The core challenge is to decouple concept learning from reasoning learning, transferring the systematic "syntax" of reasoning from symbolic programs into the semantic flexibility of MLLMs.

2. Methodology: APEIRIA

The authors introduce APEIRIA (Apeiria, meaning "Unlimited" in Greek), a neuro-symbolic 3D MLLM that bridges these paradigms by distilling symbolic reasoning patterns into natural language Chain-of-Thought (CoT). The model is trained via a three-stage curriculum:

3.1. Object-Centric Representation

APEIRIA utilizes an efficient object-centric scene representation to minimize input tokens (approx. 400 tokens vs. 10k–40k in video-based MLLMs).

• Segmentation: Scenes are segmented into object instances using Mask3D.
• Feature Extraction: Visual-geometric features are extracted via Uni3D, and 2D appearance features via DINOv2.
• Spatial Encoding: Object locations and dimensions are encoded via learnable positional embeddings.
• Input: These features are concatenated and interleaved with instruction tokens as input to the LLM.

3.2. Curriculum-Based Reasoning Distillation

The training process progressively builds reasoning capabilities through three stages:

Stage 1: Perception Alignment

• Goal: Ground 3D visual-geometric features to the LLM's textual embedding space.
• Method: Standard 3D vision-language pre-training on a mixture of object-centric tasks (recognition, localization, and captioning) using ~193K instruction-response pairs from datasets like ScanNet and MMScan.
• Outcome: The model learns to recognize object categories, attributes, and locations, establishing a foundation for scene understanding.

Stage 2: Symbolic Reasoning Injection (CoT-SFT)

• Goal: Teach the model the "syntax" of spatial reasoning (iterative decomposition and verification).
• Method: The model is fine-tuned using Chain-of-Thought Supervised Fine-Tuning (CoT-SFT) on traces derived from neuro-symbolic programs.
  • Data Construction: Programs from synthetic datasets (Sr3D) and annotated datasets (ScanNet, MMScan) are parsed into execution traces. Each step is serialized into natural language containing:
    • Plans: Explicit reasoning goals (e.g., "Find all objects of category 'vase'").
    • Executions: Step-by-step inputs and outputs with object details (IDs, positions, sizes).
  • Trace Structure: The CoT explicitly maps symbolic operations (e.g., filter, relate) to transparent reasoning steps, ensuring objects are referenced by unique IDs and spatial configurations.
• Outcome: The model learns systematic query decomposition and stepwise spatial verification, internalizing the reasoning patterns of NS3D without relying on rigid concept-specific networks.

Stage 3: Open-Set and Complex Reasoning Generalization (CoT-RL)

• Goal: Extend learned reasoning patterns to real-world scenarios involving open-vocabulary concepts and deep nesting where step-by-step supervision is unavailable.
• Method: Chain-of-Thought Reinforcement Learning (CoT-RL) using Group Relative Policy Optimization (GRPO).
  • Reward Function: A composite reward is used to guide exploration:
    • Soft Grounding Reward: Evaluates configuration similarity (location and size) between predicted and ground-truth objects to provide dense feedback, avoiding the sparsity of standard IoU metrics.
    • Format Reward: A binary constraint ensuring the model maintains the structured CoT format (plan and thinking tags) to prevent "structure collapse" into direct answering.
• Outcome: The model adapts to complex, nested instructions and open-vocabulary descriptors (e.g., "comfortable furniture") while retaining the systematic reasoning structure.

3.3. Modular Inference Enhancement

A unique feature of APEIRIA is the explicit decoupling of planning (reasoning) and execution (perception).

• External Plan Injection: The reasoning trace can accept plans generated by external, more powerful LLMs (e.g., GPT-4, Claude) without retraining.
• Perception Module Replacement: The scene() primitive execution can be swapped with outputs from state-of-the-art 3D perception models (e.g., SegDINO3D) by formatting their output to match the CoT trace. This allows the system to leverage improvements in perception models dynamically.

3. Key Contributions

1. APEIRIA Architecture: A neuro-symbolic 3D MLLM that unifies the systematic rigor of symbolic methods with the semantic flexibility of LLMs by modeling reasoning as transparent Chain-of-Thought.
2. Curriculum-Based Distillation Framework: A novel three-stage training pipeline that progressively transfers reasoning patterns from neuro-symbolic programs to 3D MLLMs: from perception alignment, through symbolic reasoning injection, to open-set generalization via reinforcement learning.
3. Empirical Validation: Demonstration that APEIRIA surpasses prior neuro-symbolic methods and matches or exceeds state-of-the-art 3D MLLMs on 3D spatial reasoning benchmarks, while maintaining transparent reasoning and efficient generalization.

4. Experimental Results

Evaluations were conducted on ScanRefer and Multi3DRefer for spatial grounding, SQA3D for situated question answering, and Scan2Cap for dense captioning.

• Spatial Reasoning Performance: APEIRIA achieved 58.4% Acc@0.25 and 51.2% Acc@0.5 on ScanRefer, and 59.2% F1@0.25 and 53.8% F1@0.5 on Multi3DRefer. With modular inference enhancement (using external planners/perception), these scores improved to 60.5% and 53.2% (ScanRefer) and 60.9% and 55.2% (Multi3DRefer), outperforming all baselines including Chat-Scene and Inst3D-LMM.
• Zero-Shot Generalization: In a zero-shot setting on Nr3D (trained only on synthetic Sr3D data), APEIRIA achieved 36.5% accuracy, surpassing the fully supervised NS3D baseline (33.9%) by 2.6%. This demonstrates the model's ability to generalize to open-vocabulary concepts ("messy," "cozy") that traditional NS3D methods cannot handle.
• Task Generalization: The framework successfully transferred to SQA3D and Scan2Cap without modifying the symbolic reasoning pipeline, only changing the outcome reward, confirming the paradigm's generality.
• Ablation Studies:
  • Removing Stage 3 (RL) caused a significant performance drop (6.9% on ScanRefer), confirming RL's necessity for adapting to complex real-world instructions.
  • Removing Stage 2 (CoT-SFT) led to catastrophic failure, proving the need for structured "warm start" reasoning.
  • Removing the "Thinking" trace (forcing direct answers) reduced performance, indicating that the explicit reasoning procedure is essential for unlocking full spatial reasoning potential.
  • Reasoning complexity analysis showed that RL benefits increase with the number of reasoning steps, providing supervision for complex chains where SFT lacks traces.

5. Significance and Claims

The paper claims that APEIRIA successfully bridges the gap between interpretable but limited symbolic methods and flexible but opaque end-to-end MLLMs. By distilling reasoning patterns rather than concept-specific knowledge, the method preserves key neuro-symbolic virtues:

• Transparency: Inference is transparent through explicit reasoning traces, allowing for failure diagnosis and debugging.
• Modularity: The separation of planning and perception allows for plug-and-play integration of stronger components (e.g., better LLMs or 3D detectors) without retraining.
• Scalability: The approach enables the transfer of systematic reasoning capabilities from synthetic, fully supervised environments to real-world, open-vocabulary scenarios.

The authors position this work as an initial stepstone toward interpretable and useful embodied agents, offering a framework for systematic and verifiable grounding in 3D environments. They acknowledge limitations, noting that performance is still bounded by the underlying 3D perception module and that very deep reasoning chains or physics-based reasoning remain areas for future exploration.