Technical Summary: Intern-S2-Mobius: Foundation Model with Decoupled Knowledge and Reasoning

1. Problem Statement

The development of Foundation Models is currently facing a dual bottleneck. First, the prevailing Scaling Law is approaching diminishing marginal returns; simply increasing model parameters, training data, or reasoning chain lengths yields progressively smaller capability gains at exponentially higher costs. Second, while long chains of thought (CoT) have improved performance in domains like mathematics and code, they often result in verbose, redundant outputs that degrade inference efficiency.

Conversely, attempts to reduce architectural complexity (e.g., linear attention, SSMs) to improve efficiency have often come at the cost of significant capability loss, rendering them unsuitable for commercial deployment. The core issue identified is the tight coupling of knowledge storage (Feed-Forward Networks) and reasoning operators (Self-Attention) within the hierarchical, unidirectional structure of the Transformer. This coupling forces models to rely on token-mediated, iterative generation to access deep-layer knowledge, leading to inefficient computation and excessive token redundancy.

2. Methodology: The Mobius Architecture

To address these limitations, the authors propose Mobius, an architecture that decouples knowledge and reasoning. The design is built upon two primary innovations:

2.1. Decoupled Knowledge and Reasoning

Unlike the Transformer, where Feed-Forward Networks (FFN) are bound to specific layers, Mobius introduces:

• Globally Shared Memory (FFN): A single, massive knowledge vector database shared across all layers, constructed via horizontal concatenation (and block-wise partitioning analogous to MoE at larger scales) of knowledge vectors. This acts as a repository for knowledge vectors.
• Multiple Reasoners (Self-Attention): Several Self-Attention modules that iteratively query the shared memory.
• Mechanism: Reasoners use hidden states as a cache and carrier, repeatedly querying the memory for required knowledge vectors. Knowledge is transmitted back to the reasoning operators, allowing for compositional reasoning without the rigid layer-by-layer constraints of Transformers.

2.2. Innate Architectural Talents

The decoupling enables two specific capabilities:

1. Backward Residual Connection: In standard Transformers, residual connections are unidirectional (shallow to deep). Mobius's shared memory allows deep layers to access shallow-layer knowledge (and vice versa) during forward propagation. This is achieved indirectly by granting all layers access to the same global repository, enhancing compositional generalization and accelerating information synthesis.
2. Dynamic Latent Reasoning: Mobius internalizes deliberation, trial-and-error, and refinement into the optimization of continuous vectors rather than discrete tokens.
   • Higher Loop Frequency: Iterations occur within significantly fewer layers compared to traditional latent reasoning.
   • High-Density Latents: The model acquires joint representations of multiple tokens simultaneously via compact, continuous vectors.
   • Dynamic Allocation: Computational budget is dynamically allocated to different tokens, allowing the model to decode multiple tokens in a single burst after refining high-information-density latents.

2.3. Implementation Details

• Knowledge Repository Construction: The FFN is constructed via horizontal concatenation of knowledge vectors. To manage parameter scale and activation efficiency, the authors employ a block-wise partitioning technique (analogous to MoE) with sparse activation during forward propagation.
• Training Strategy: The authors evaluated Mobius via two paths:
  1. Training-from-Scratch (TFS): A 7B model trained on 1TB tokens.
  2. Continual Pre-Training (CPT): Starting from the Qwen3.5-35B checkpoint, continuing pre-training on 1TB tokens, followed by Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL).

3. Key Results

3.1. Data Efficiency (Training-from-Scratch)

• A 7B Mobius model trained from scratch achieved MMLU scores comparable to a 7B Transformer baseline.
• Crucially, Mobius reached this performance using only 62.6% of the training data required by the Transformer baseline, demonstrating a 1.6× improvement in data efficiency.

3.2. Inference Efficiency and Performance (Continual Pre-Training)

• Performance Parity: The Intern-S2-Mobius-35B (continually pre-trained from Qwen3.5-35B) achieved similar or superior downstream scores compared to the Qwen3.5-35B baseline across general and scientific benchmarks (e.g., MMLU Pro, GPQA Diamond, IMO Bench, AIME 2026).
• Inference Speedup: Intern-S2-Mobius delivered nearly a 4× end-to-end inference speedup compared to the Transformer baseline with the same parameter count.
• Reduced Output Length: The speedup is primarily driven by significantly shorter reasoning chains. On average, Mobius generated 1.2× to 5.0× fewer tokens than the Transformer baseline across various benchmarks (e.g., 4.6× shorter on MMLU Pro).
• Token Efficiency: In a step-aligned comparison on a linear algebra problem, Mobius completed the same reasoning steps with 516 tokens compared to 2,364 tokens for the Transformer, primarily by eliminating repeated derivations and redundant checks.

4. Significance and Claims

The paper claims that Mobius represents an alternative path to scaling AI: increasing architectural complexity to raise the ceiling of model intelligence, thereby reducing end-to-end overhead.

• Efficiency via Density: By shifting from token-based reasoning to latent-based, high-density reasoning, Mobius achieves substantial efficiency gains without sacrificing capability.
• Bidirectional Knowledge Access: The introduction of "Backward Residual Connection" via shared memory solves the unidirectional information flow limitation of Transformers, allowing for more flexible and efficient knowledge retrieval.
• Future Potential: The authors posit that the decoupling of knowledge and reasoning offers preliminary characteristics that could support:
  • Self-Evolving Systems: Enabling models to acquire new skills without catastrophic forgetting of previous knowledge, though joint design of foundation models, agentic systems, and environmental feedback is required for validation.
  • World Models: Offering a stronger prior for modeling continuous-space inputs compared to Transformers, though the latent reasoning prior "may prove far from sufficient" and likely requires reengineering the attention mechanism.
  • Scientific Discovery: Facilitating the composition and generalization of knowledge across domains to generate disruptive hypotheses, though real-world performance requires joint optimization of data, infrastructure, and algorithms.
  • Hardware-Software Co-Design: Opening the possibility of a deployment strategy where reasoning parameters reside in GPU memory while knowledge parameters are stored on SSDs and loaded on demand. However, the authors note this vision "remains to be realized" and requires "substantial effort."

The paper concludes that while Mobius demonstrates promising results in efficiency and capability, its full potential in real-world scenarios (self-evolution, world modeling, scientific discovery) requires further joint optimization of data, infrastructure, and training algorithms.