Technical Summary: CoLT – Chain of Latent Thoughts

1. Problem Statement

While Chain-of-Thought (CoT) reasoning has significantly enhanced the capabilities of Multi-Modal Large Language Models (MLLMs) in complex visual tasks, the prevailing text-based paradigm faces three critical limitations:

1. Inference Latency: Generating lengthy reasoning chains in natural language is autoregressive and token-intensive, often consuming hundreds or thousands of tokens, which constitutes the primary computational bottleneck during inference.
2. Expressiveness Constraints: Text-based reasoning is inherently deterministic. Once a reasoning step is committed to a specific natural language form, it collapses the rich distribution of potential solving trajectories into a single surface realization, eliminating alternative reasoning pathways.
3. Error Propagation: Errors made in early reasoning steps can propagate and compound through subsequent steps, a phenomenon particularly detrimental in long reasoning chains.

Existing attempts at "latent reasoning" (reasoning in continuous representation spaces) have struggled with training instability and the generation of semantically meaningless vectors. Furthermore, recent latent visual reasoning approaches that rely on auxiliary images for supervision introduce high annotation costs and scalability issues.

2. Methodology: CoLT Framework

The authors propose CoLT (Chain of Latent Thoughts), a framework that enables MLLMs to reason through a chain of latent thought representations ($h_1, \dots, h_K$) rather than verbose text tokens. The framework replaces the standard text generation process with $K$ discrete latent steps (where $K \ll L$, the number of text tokens), bypassing the text generation bottleneck while preserving multi-path exploration in the latent space.

To address the challenge of training stable and meaningful latent reasoning, CoLT introduces a comprehensive supervision strategy comprising three complementary mechanisms:

A. Latent Thought Representation

Instead of generating text tokens, the model produces hidden state vectors at designated reasoning positions. These vectors are fed back into the language model as input embeddings for subsequent steps, maintaining the autoregressive nature of the model without extending the vocabulary.

B. External Decoder Supervision

A lightweight external decoder ($D_\phi$), sharing the same tokenizer and vocabulary as the backbone model, provides step-level supervision in two directions:

• Forward Decoding: The decoder takes the preceding latent thought $h_k$ as input and is trained to decode the textual reasoning content of the next step ($r_{k+1}$). This ensures latent thoughts encode sufficient information to reconstruct explicit reasoning.
• Backward Decoding: The decoder receives preceding textual reasoning steps ($r_{k-1}$) and produces hidden states ($\tilde{h}_k$) that are aligned with the model's latent thought $h_k$. This anchors the latent space to the semantic structure of explicit reasoning chains.

C. Internal Step-Level Supervision

A lightweight projection head ($f_\theta$) encourages coherent transitions between consecutive latent steps. It predicts the next step's representation ($\hat{h}_{k+1}$) from the current one ($h_k$), regularizing the latent chain through step-to-step consistency.

D. Training Objective

The total loss function combines the standard task loss (generating the final answer) with the three supervision signals:
$$\mathcal{L} = \mathcal{L}_{task} + \alpha \mathcal{L}_{fwd} + \beta \mathcal{L}_{bwd} + \gamma \mathcal{L}_{int}$$
The training proceeds in two stages:

1. Supervised Fine-Tuning (SFT): The backbone and decoder are trained on CoT annotations (from the OneThinker dataset) using the combined loss to learn structured latent thoughts.
2. Reinforcement Learning (RL): Group Relative Policy Optimization (GRPO) is applied on CoT samples. In this stage, the auxiliary decoder and internal supervision are removed, allowing the model to update latent thoughts freely based on outcome-based rewards.

During inference, the external decoder and projection head are discarded, incurring no additional computational overhead.

3. Key Contributions

• Novel Framework: Introduction of CoLT, which teaches MLLMs to reason via latent vectors rather than text tokens, achieving effective reasoning with as few as 3 steps.
• Stabilization Mechanism: A principled supervision framework using a lightweight external decoder (forward and backward modes) and internal prediction to prevent the collapse of latent representations into meaningless semantics.
• Annotation Efficiency: Unlike latent visual reasoning methods, CoLT operates entirely within the language model's latent space, requiring no auxiliary image annotations or specialized encoders.
• Efficiency: The method eliminates the need for verbose text generation during the reasoning phase, significantly reducing inference time.

4. Experimental Results

Experiments were conducted on eight diverse multi-modal benchmarks (SeedBench, MMBench, ChartQA, TextVQA, ScienceQA, MMStar, AI2D, and MMT-Bench) using a Qwen3-VL-8B backbone.

• Performance: CoLT outperforms existing latent reasoning methods (e.g., CODI, SIM-CoT) and latent visual reasoning approaches (e.g., LVR). Specifically, it surpasses the strongest latent baseline (SIM-CoT) by +5.1% in average accuracy and the strongest latent visual baseline (LVR) by +5.5%.
• Comparison with Text CoT: CoLT achieves +3.4% higher average accuracy compared to text-based CoT, with significant gains on ChartQA (+9.6%) and TextVQA (+6.1%).
• Efficiency: CoLT reduces inference time by 10.1× and text decoding time by 22.6× compared to text CoT methods, while maintaining higher accuracy.
• Robustness: Under visual noise (Gaussian blur, occlusion) and text noise (spelling errors, key deletion), CoLT demonstrates superior robustness, with accuracy degradation significantly lower than both Direct Answer and Text CoT baselines.
• Ablation Studies:
  • All three supervision signals (forward, backward, internal) are non-redundant; the full combination yields optimal performance.
  • A lightweight 0.6B decoder is sufficient for effective supervision, with larger decoders providing negligible gains.
  • The optimal number of latent steps is $K=3$; performance degrades slightly with excessive steps.

5. Significance and Claims

The paper claims that CoLT represents a fundamental shift in how multi-modal models perform reasoning. By moving from explicit text generation to latent thought vectors, CoLT addresses the inherent inefficiency and expressiveness limits of text-based CoT. The authors emphasize that their method achieves superior efficiency (up to 22.6× speedup) without sacrificing, and often improving, reasoning quality.

The significance lies in the ability to perform "thinking" in a continuous space that is both interpretable (verified via forward decoding visualization) and robust to input noise. The framework demonstrates that complex visual reasoning can be internalized into compact latent representations, removing the dependency on costly auxiliary annotations and the latency of token-by-token generation. The authors conclude that this approach offers a scalable path toward high-intelligence agents that can reason efficiently.