💬 NLP

Scaling Embeddings Outperforms Scaling Experts in Language Models

This paper demonstrates that scaling embedding dimensions can be a more efficient way to achieve sparsity and performance gains than scaling Mixture-of-Experts (MoE), a finding validated by the introduction of LongCat-Flash-Lite, a high-performing 68.5B parameter model with minimal activation costs.

Original authors: Hong Liu, Jiaqi Zhang, Chao Wang, Xing Hu, Linkun Lyu, Jiaqi Sun, Xurui Yang, Bo Wang, Fengcun Li, Yulei Qian, Lingtong Si, Yerui Sun, Rumei Li, Peng Pei, Yuchen Xie, Xunliang Cai

Published 2026-02-12

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Hong Liu, Jiaqi Zhang, Chao Wang, Xing Hu, Linkun Lyu, Jiaqi Sun, Xurui Yang, Bo Wang, Fengcun Li, Yulei Qian, Lingtong Si, Yerui Sun, Rumei Li, Peng Pei, Yuchen Xie, Xunliang Cai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a massive, high-tech library. To make this library smarter, you have two main ways to expand it:

The "Expert" Method (Scaling Experts): You hire thousands of specialized professors (Experts). If a student asks about biology, you call the biology professor. If they ask about history, you call the history professor.
The "Vocabulary" Method (Scaling Embeddings): Instead of hiring more professors, you give every single book in the library a much more detailed, hyper-specific index. Instead of just a label that says "Apple," the index says "Crisp, red, sweet, autumnal, fruit, Granny Smith."

For a long time, the AI world has been obsessed with the Expert Method. We keep adding more and more professors to handle more topics. But the researchers at Meituan found a problem: eventually, hiring more professors becomes too expensive, they start bumping into each other in the hallways (system bottlenecks), and they don't actually make the library much smarter.

This paper proposes a "Third Way": Scaling the Embeddings.

The Core Idea: The "N-gram" Super-Index

The researchers introduced something called N-gram Embedding.

Think of a standard AI like a person who reads one word at a time. They see "New," then "York," then "City." They understand it, but they are processing it piece by piece.

The N-gram method is like giving that person "Contextual Vision." Instead of just seeing individual words, the AI sees "chunks" of meaning. It doesn't just see "New"; it sees "New York" as a single, powerful concept. It sees "New York City" as an even deeper concept. By expanding the "index" (the embeddings) to include these multi-word chunks, the AI gains a massive amount of knowledge without needing to hire more "professors" (experts) to do the heavy lifting.

The "Sweet Spot" (The Goldilocks Principle)

The researchers discovered that you can't just do this infinitely. They found a "Goldilocks Zone":

Too little embedding scaling: You’re just a standard AI; you miss out on the deep context.
Too much embedding scaling: You spend so much money on the "index" that you don't have enough money left to hire any professors at all! The library becomes a giant list of definitions with no one to explain them.
The Sweet Spot: They found that if you allocate about 50% of your "budget" to these super-detailed embeddings, you get a much smarter model than if you had spent that same money on more experts.

The Result: LongCat-Flash-Lite

To prove this works, they built a model called LongCat-Flash-Lite.

It is a massive model (68.5 billion parameters), but it is incredibly "lean." Even though it is huge, it only "wakes up" a tiny fraction of its brain (about 3 to 4.5 billion parameters) to answer any single question.

Because they spent so much of their budget on the "Super-Index" (the embeddings) rather than the "Professors" (the experts), the model is:

Smarter at specialized tasks: It is exceptionally good at coding and acting as an AI Agent (doing tasks like using tools or navigating a computer).
Faster: Because it doesn't have to "call" as many professors to get an answer, it can respond much more quickly.

Summary in a Nutshell

Instead of trying to build a smarter brain by adding more "thinking modules" (which gets slow and messy), this paper shows you can build a smarter brain by giving it a much more sophisticated way to perceive the world (better embeddings). It’s the difference between hiring more people to study a map and simply giving the person holding the map a much higher-resolution version of the map.

Technical Summary: Scaling Embeddings Outperforms Scaling Experts in Language Models

1. Problem Statement

The current paradigm for scaling Large Language Models (LLMs) relies heavily on Mixture-of-Experts (MoE) architectures. While MoE allows for massive parameter counts with manageable computation, it faces two major limitations:

Diminishing Returns: As models scale, the marginal performance gain from adding more experts decreases (efficiency saturation).
System Bottlenecks: Increasing the number of experts exacerbates communication overhead and memory bandwidth pressure in distributed training and inference.

The authors identify the embedding layer as an overlooked, inherently sparse dimension that offers $O(1)$ lookup complexity. They seek to determine if scaling embeddings can serve as an orthogonal and more efficient dimension for model scaling compared to scaling experts.

2. Methodology

The researchers propose and extensively test the N-gram Embedding layer as a method for parameter expansion.

N-gram Embedding Architecture: Instead of relying solely on a base vocabulary, the model augments token representations by looking up $n$ -gram sequences. It uses a polynomial rolling hash function to map $n$ -grams to sub-tables. To mitigate hash collisions, they decompose embedding tables into $K$ sub-tables with different vocabulary sizes and apply linear projections to map them back to the hidden dimension.
Comparative Scaling Framework: They conducted from-scratch pre-training experiments (on 300B tokens) comparing N-gram Embedding (NE) scaling against parameter-equivalent MoE scaling (where the same parameter budget is used to add more experts).
Architectural Optimization:
- Embedding Amplification: To prevent the embedding signal from being "drowned out" by the residual stream (as attention outputs often have much larger L2 norms), they applied scaling factors ( $\sqrt{D}$ ) and LayerNorm to the embedding output.
- System-Level Optimizations: They developed a specialized N-gram Cache and custom CUDA kernels to handle the increased I/O of large embedding tables. They also integrated speculative decoding to convert parameter sparsity into actual inference speedups.

3. Key Contributions

Discovery of Scaling Regimes: They identified that embedding scaling is most effective when the model is already at a high sparsity level (high total-to-activated parameter ratio).
Architectural Design Principles:
1. Budgeting: Do not allocate more than 50% of the total parameter budget to N-gram embeddings.
2. Collision Avoidance: N-gram vocabulary sizes should significantly deviate from integer multiples of the base vocabulary to prevent hash collisions.
3. Width vs. Depth: Increasing model width enhances the advantage of N-gram embeddings, whereas increasing depth diminishes it.
LongCat-Flash-Lite: The introduction of a 68.5B parameter model that uses ~30B parameters for embeddings and only ~3B–4.5B activated parameters.

4. Results

Superior Pareto Frontier: In high-sparsity regimes, N-gram Embedding achieves lower training and validation loss than MoE models with an equivalent total parameter count.
LongCat-Flash-Lite Performance:
- Base Model: Outperforms its "Vanilla" MoE counterpart (which uses the same parameters but as experts) across general, reasoning, and coding benchmarks.
- Agentic & Coding Excellence: The model shows exceptional strength in Agentic Tool Use (e.g., $\tau^2$ -Bench) and Agentic Coding (e.g., SWE-Bench), where it significantly outperforms competitors like Gemini 2.5 Flash-Lite and Qwen3-Next.
- Inference Efficiency: Through kernel fusion and speculative decoding, the model achieves high throughput despite its massive embedding size.

5. Significance

This paper shifts the perspective on LLM scaling. It proves that parameter capacity does not have to be synonymous with expert capacity. By shifting the parameter budget from the computationally expensive FFN/MoE layers to the computationally "cheap" (O(1) lookup) embedding layer, researchers can build models that are both more intelligent (better semantic representation) and more efficient (lower active parameter count). This provides a new roadmap for developing massive-scale, sparse models that are optimized for real-world agentic and coding applications.