💬 NLP

Index SLM Technical Report

Bilibili introduces Index-1.9B, a series of open small language models featuring a 1.9B-parameter foundation trained on 2.8 trillion tokens with a specialized Warmup-Stable-Decay schedule and Norm-Head layer, which achieves competitive performance on standard benchmarks and includes variants for pure pre-training, chat alignment, and character-based role-playing.

Original authors: Lusheng Zhang, Shien He, Tianxing Yan, Mengran Yu, Ziang Cui, Kai Zhao, Xiaojing Liu, Tianjiao Li

Published 2026-07-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Lusheng Zhang, Shien He, Tianxing Yan, Mengran Yu, Ziang Cui, Kai Zhao, Xiaojing Liu, Tianjiao Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're building a tiny, super-smart robot brain that fits in your pocket. Most people think bigger brains are always better, but a team at Bilibili decided to see if they could make a 1.9-billion-parameter robot (let's call it Index-1.9B) that punches way above its weight class. They didn't just shrink a giant model; they rebuilt the engine from the ground up using 2.8 trillion words of reading material, mostly in Chinese and English.

Here's the story of how they did it, what they discovered, and the weird surprises they found along the way.

The Secret Sauce: How They Trained the Brain

1. The Reading List (Data)
Think of the training data as the robot's library. They didn't just dump the whole internet in there. They cleaned it up like a librarian organizing a messy attic. They removed duplicates (even finding a menu of month names that was copied 156,000 times by mistake!) and filtered out bias.

The Mix: The library is mostly books and web pages, but they made sure to include 6% code and 10% high-quality stuff like academic papers and encyclopedias.
The Surprise: They found that reading "instruction manuals" (like "Write a poem" or "Solve this math problem") during the final stage of training made the robot look much smarter on tests. In fact, adding just 7% of this instruction data boosted its test scores by about 7 points. They even released two versions of the robot—one with this extra reading and one without—so everyone could see exactly how much it helped.

2. The Brain Structure (Architecture)
Usually, when you have a fixed amount of "brain power" (parameters), you can make the brain wide (lots of neurons in one layer) or deep (many layers stacked up).

The Finding: The team tried a "deep and narrow" design (36 layers) versus a "wide and shallow" one (9 layers). The deep one won every time. It's like building a skyscraper instead of a wide, flat warehouse; for this size, going up worked better than going out.
The Stabilizer: They also added a special "Norm-Head" layer. Imagine the robot's brain getting a little dizzy when it tries to guess the next word because some words are rare. This new layer acts like a stabilizer, keeping the brain calm even when the robot is learning super fast.

3. The Learning Schedule (The "WSD" Method)
Most robots learn at a steady pace or slow down gradually. Index-1.9B uses a "Warmup–Stable–Decay" (WSD) schedule.

The Strategy: First, it warms up. Then, it learns at a constant, high speed (the "Stable" phase) while reading the whole library. Finally, in the "Decay" phase, it slows down but switches to reading only the best, most curated books (the academic papers and encyclopedias).
The Result: This combination of slowing down while reading high-quality data gave it the biggest boost in performance.

The Weird Surprise: The "Surge"

Here is the part that even the authors can't fully explain yet.
During the "Stable" phase, when the robot was reading at a constant speed, something strange happened. Between 1.0 and 1.2 trillion tokens of reading, the robot's test scores suddenly jumped up. It went from being average to surpassing a number of 7B models, all without changing the learning speed or the type of data.

The Certainty: The paper admits this is a mystery. They suggest that maybe the high-quality data combined with the fast learning rate just clicked at that moment, but they haven't proved exactly why. It's like a student suddenly understanding everything after reading a specific chapter, even though the teacher hasn't changed the lesson plan.

The Different Versions of the Robot

The team didn't just stop at one model. They released a whole family:

Index-1.9B-Base: The raw, smart brain ready for anything.
Index-1.9B-Pure: A control version that never read instruction manuals. This proves that the "smart" scores of the main model really do come from that extra reading.
Index-1.9B-Chat: The Base model taught to talk nicely to humans, using a technique called "Direct Preference Optimization" (DPO). This is like teaching the robot not just to answer, but to answer well by showing it examples of good vs. bad conversations.
Index-1.9B-Character: A version that can role-play. It uses a "retrieval" trick, where it looks up past conversations with a specific character to stay in character. It's so good at this that it ranks higher than many much larger models on role-playing tests.

What It Can (and Can't) Do

The Wins:

Smarts: On standard tests for math, reasoning, and general knowledge, Index-1.9B scores an average of 64.92. This is competitive with, and sometimes beats, models that are several times bigger. Specifically, it surpasses the Llama-2-13B model on the overall average score, though it doesn't beat every single benchmark against every larger model.
Language: It's great at Chinese and English. Additionally, the tokenizer they built for the model achieves the strongest compression rates for Japanese and Korean among the tokenizers they compared, making it efficient for these languages.
Role-Playing: It can act out characters with high consistency, ranking 9th overall on a major leaderboard, beating many 7B to 14B models.

The Limits:

Math and Code: While it's decent, the authors admit these are still areas for improvement. It's not a math genius yet.
Facts: Because it's small, it might still make up facts or misunderstand complex instructions.
The Mystery: That sudden performance jump during training? They still don't know exactly why it happened.

The Bottom Line

The paper shows that you don't need a massive, expensive brain to be smart. If you feed a small robot the right mix of high-quality books, teach it to read deeply rather than broadly, and give it a special stabilizer for its brain, it can compete with giants. They also proved that "instruction data" (learning how to follow orders) is a huge cheat code for test scores, and they released the evidence so no one can hide the fact.

It's a small model with a big personality, a few open questions, and a whole lot of potential.

Technical Summary: Index-1.9B Technical Report

Problem and Motivation

Small Language Models (SLMs) with 1–3 billion parameters have recently demonstrated capabilities previously reserved for much larger models, driven by higher-quality data and training beyond compute-optimal token budgets. While these models offer the advantage of modest inference costs suitable for large-scale and on-device deployment, the field lacks transparency regarding specific design choices, particularly concerning the interaction between learning-rate schedules, data quality, and the inclusion of instruction data during pre-training. This report addresses these gaps by presenting Index-1.9B, a series of open SLMs developed at Bilibili, and providing a suite of controlled experiments to demystify the training dynamics of models in this parameter range.

Methodology

Data and Pre-training

The Index-1.9B series is pre-trained on 2.8 trillion tokens, with a Chinese-to-English ratio of 4:5 and 6% code. The corpus includes web pages, in-house data, code, academic papers, and curated domain-specific materials (books, encyclopedias).

Data Cleaning: The team employed bias-aware filtering, document-level deduplication (using MinHash on tens of billions of documents), and exact substring deduplication to remove repetitive text patterns (e.g., menu strings) that standard paragraph-level deduplication misses.
Tokenizer: A custom Byte-Pair Encoding (BPE) tokenizer was trained with a vocabulary size of 65,029. Key modifications include training the Chinese sub-vocabulary separately with a reduced maximum piece length (5 vs. 16) and removing the automatic whitespace prepending found in Llama-family tokenizers to better suit Chinese text.

Model Architecture

Index-1.9B follows a decoder-only Transformer design with Llama-style conventions (RoPE, SwiGLU, RMSNorm) but introduces two specific modifications:

Depth: The model utilizes 36 layers (hidden size 2048, FFN size 5888, 16 heads). Experiments indicated that at a fixed parameter count, increasing depth improves downstream performance more than increasing width.
Norm-Head: To stabilize training under large learning rates, the output projection layer (LM head) employs Norm-Head, which normalizes head weights to dynamically rescale them. This addresses the issue where the LM head gradient is an order of magnitude larger than other layers, causing oscillation on rare tokens.

Training Recipe

The training follows a Warmup–Stable–Decay (WSD) learning-rate schedule:

Stable Phase: A constant learning rate ( $5 \times 10^{-4}$ ) is maintained while training on the global data mixture.
Decay Phase: The learning rate decays to $5 \times 10^{-6}$ (a 100x range). Crucially, during this phase, the concentration of curated data (high-quality books, papers, etc.) is substantially raised.
Instruction Data: The report explicitly investigates the impact of instruction data. A control variant, Index-1.9B-Pure, is trained with all instruction-like data strictly filtered from the corpus, while the main Index-1.9B (Boost) includes a proportion of instruction data during the decay phase.

Alignment and Specialization

Index-1.9B-Chat: Aligned via Supervised Fine-Tuning (SFT) on ~100k high-quality instruction pairs, followed by Direct Preference Optimization (DPO) on writing, instruction following, and safety. For safety, the team utilized self-generated refusals rather than human-written ones to avoid catastrophic forgetting.
Index-1.9B-Character: Extends the chat model with Retrieval-Augmented Generation (RAG) for few-shot role-playing, retrieving relevant character dialogues to condition the generation.

Key Contributions and Controlled Studies

Beyond the model weights, the paper's primary contribution is a set of transparent, controlled ablation studies:

Norm-Head Efficacy: Demonstrates that Norm-Head stabilizes training and allows for higher learning rates, yielding consistent benchmark gains.
Depth vs. Width: Confirms that for a fixed parameter count (1.01B), a deeper (36-layer) model outperforms a wider, shallower (9-layer) counterpart.
Learning-Rate Schedules & Data Quality: Shows that while different schedules (Cosine, Linear, WSD) converge to similar final losses, the WSD schedule combined with high-curated data during the decay phase yields the best benchmark scores.
Instruction Data Impact: Quantifies the "inflation" of benchmark scores caused by including instruction data during pre-training. The study reveals that adding just 7% instruction data during the decay phase can increase MMLU scores by approximately 7 points, significantly altering leaderboard rankings.
Performance Surge: Documents an unexplained, abrupt surge in benchmark performance (C-Eval and MMLU) occurring midway through the constant-learning-rate phase (between 1.0T and 1.2T tokens), prior to any learning-rate decay.

Results

Base Model Performance: Index-1.9B achieves an average benchmark score of 64.92 across six tasks (MMLU, C-Eval, CMMLU, HellaSwag, ARC-C, ARC-E). This performance is competitive with or exceeds open models several times its size (e.g., outperforming Llama-2-13B on average) and is surpassed only by Qwen2-1.5B among comparable 1.5B–2B models.
Chat Model: The aligned model shows significant improvements in open-ended QA and writing tasks via DPO, with strong conversational behavior attributed to the pre-training on community-forum corpora.
Role-Playing: Index-1.9B-Character ranks ninth on the CharacterEval leaderboard, outperforming all other models of comparable scale (1.9B) and competing with 7B–14B models.
Math and Code: These remain areas for improvement, with scores on par with Llama-2-13B but lower than specialized larger models.

Significance and Claims

The authors claim that Index-1.9B demonstrates that small models can achieve high performance through rigorous data curation, architectural adjustments (Norm-Head, depth), and optimized training schedules (WSD with data mixing).

The report emphasizes transparency as a core contribution. By releasing the Index-1.9B-Pure (no instruction data) alongside the standard model, and by detailing the specific impact of instruction data on benchmarks, the authors aim to make comparisons in the 1B–3B parameter class more honest and reproducible. They explicitly note that the inclusion of instruction data during pre-training can artificially inflate benchmark scores, a factor often obscured in public reports.

Finally, the paper highlights the "performance surge" observed during the stable phase as an unexplained phenomenon worthy of future investigation, suggesting that high-quality data combined with stable, large learning rates may allow models to reach strong performance milestones before the decay phase begins. All models, code, and evaluation scripts are released for academic and commercial use.