💬 NLP

Llamion Technical Report

The paper introduces Llamion, a family of 14B-parameter open-weight models that successfully transforms the Orion-14B architecture into the Llama format using a novel recipe called KEPT, achieving state-of-the-art performance on benchmarks like KoMMLU and preserving advanced capabilities such as long-context handling with minimal training resources.

Original authors: Kisu Yang, Yoonna Jang, Hyeonseok Moon, Hwanseok Jang, Taewoo Lee, Hyungjin Lee, Jeseung Lee, Juhyoung Park, Heuiseok Lim

Published 2026-05-26

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kisu Yang, Yoonna Jang, Hyeonseok Moon, Hwanseok Jang, Taewoo Lee, Hyungjin Lee, Jeseung Lee, Juhyoung Park, Heuiseok Lim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Moving a House Without Losing the Furniture

Imagine you have a magnificent, fully furnished house (a powerful AI model called Orion) that is built on a unique, custom foundation. It works great, but the local city zoning laws (the Llama architecture standard) only allow houses built on a specific, standardized foundation.

If you try to just move the house to the new foundation, the plumbing might break, the doors might not fit, and the furniture could get lost. Usually, to fix this, you would have to buy a new house and spend years re-furnishing it from scratch using old blueprints (a process called uptraining). This is expensive, slow, and requires you to have the original blueprints, which you often don't.

Llamion is the result of a new, clever moving strategy called KEPT. Instead of rebuilding the house, the team managed to move the exact same furniture and exact same layout onto the new, standardized foundation, keeping the house's personality and skills intact.

How They Did It: The "KEPT" Recipe

The team used a three-step recipe called KEPT (Efficient Knowledge Preservation for Transformation) to move the AI from Orion to the Llama style:

Normal Parameter Mapping (NPM): "The Direct Move"
For the parts of the AI that work the same way in both houses (like the vocabulary and the logic centers), they simply picked up the furniture and placed it in the exact same spot in the new house. No changes needed.
Optimized Parameter Mapping (OPM): "The Perfect Fit"
Some parts of the old house used a different type of door hinge (called LayerNorm) than the new house (which uses RMSNorm). Instead of guessing how to fix them, the team proved mathematically that if you just swap the hinges directly without any extra work, they fit perfectly. This step required zero training and no extra data. It's like realizing your old sofa fits perfectly in the new living room without needing to re-upholster it.
Cross-architecture Knowledge Distillation (XKD): "The Shadowing"
After moving the furniture, the house might feel slightly "off." To fix this, they used the original AI (Orion) as a frozen teacher. They asked the new AI (Llamion) to watch the teacher answer questions and copy its answers exactly.
- The Catch: They didn't need the original library of books the teacher was trained on. They just needed a small stack of random conversation cards (about 123 million words) to practice copying the teacher's style.

The Results: A Miracle Move

The results of this "move" were surprisingly successful:

Speed and Cost: They did this on a single powerful computer chip (an A100 GPU) in just four days. Usually, retraining a model like this takes months and costs a fortune.
Performance: The new model, Llamion, speaks Korean just as well as the original Orion. In fact, on a Korean knowledge test (KoMMLU), it scored 66.87%, beating the next-best model by a huge margin (over 7 points).
The "Magic" Transfer: The most impressive part is what they didn't teach the new model during the move.
- Coding: The training data had almost no computer code, yet Llamion can still write Python perfectly.
- Long Memory: The training data was short (4,000 words), yet Llamion can still remember and process huge documents (200,000 words) just like the original.

Why This Matters

Think of it like this: If you teach a student to mimic a master chef's cooking style using a small recipe book, the student usually only learns those specific recipes. But because Llamion was trained to mimic the master chef's brain (the hidden logic) rather than just memorize the recipes, it inherited the chef's ability to cook anything, even dishes that weren't in the small recipe book.

Summary

The paper claims that Llamion is a new AI model that takes a powerful but "non-standard" Korean AI (Orion) and transforms it into the industry-standard "Llama" format. They did this using a smart method (KEPT) that moves the model's knowledge directly and uses a tiny amount of data to fine-tune the fit. The result is a model that is compatible with standard tools, runs faster, and keeps all the original model's skills—including coding and long-memory—without needing to be retrained from scratch.

Technical Summary: Llamion and KEPT

Problem Statement

The open-source Large Language Model (LLM) ecosystem is fragmented by divergent low-level architectural choices (e.g., LayerNorm vs. RMSNorm, multi-head vs. grouped-query attention, absolute vs. RoPE positional encodings). While the field has largely converged on a standardized "Llama-family" template (decoder-only, RoPE, RMSNorm, SwiGLU), many competent pretrained models utilize non-standard architectures. Converting these models to the standard template is necessary for compatibility with mainstream tooling and deployment stacks (which often enforce trust_remote_code=False), but such architectural transformations typically result in significant knowledge loss.

The conventional remedy, uptraining (continued training on a new corpus), is costly and data-intensive. It requires a corpus that covers all previously learned knowledge to avoid catastrophic forgetting, a condition often impossible to meet when the original pretraining data is private. Furthermore, uptraining incurs a non-trivial fraction of the original pretraining compute cost.

Methodology: KEPT

The authors introduce KEPT (Efficient Knowledge Preservation for Transformation), a three-stage recipe for converting a source model ( $M_{src}$ ) with a non-standard architecture into a target model ( $M_{tgt}$ ) with a standardized architecture without requiring access to the original pretraining corpus. KEPT aligns the outputs of the modified network with the original network on any reasonable input distribution.

The process consists of:

Normal Parameter Mapping (NPM): For modules where the computation remains unchanged (e.g., token embeddings, attention projections, MLPs), parameters are copied verbatim from the source to the target. This provides a high-quality initialization.
Optimized Parameter Mapping (OPM): For modules where the computation changes but admits a principled initialization, a training-free mapping is applied. The paper focuses on the LayerNorm-to-RMSNorm transition. The authors prove that under the near-zero-mean activation regime induced by weight decay (common in models trained with AdamW), the optimal RMSNorm weights are simply the LayerNorm weights ( $\theta \leftarrow \gamma$ ), and the bias term ( $\beta$ ) can be discarded. This analytical solution eliminates the need for training on this specific module.
Cross-architecture Knowledge Distillation (XKD): A residual alignment step where the target model (student) is trained to reproduce the outputs of the frozen source model (teacher). The teacher and student are equal-sized, distinguishing this from classical compression-based distillation. The authors utilize two objectives:
- Hidden-state objective: Matches the residual stream activations layer-by-layer.
- Logit objective: Matches the final LM-head output logits. The logit objective is found to converge faster and yield marginally better results, likely because it imposes a weaker constraint on internal trajectories that are already well-initialized by NPM/OPM.

Key Contributions

Llamion Model Family: The release of three 14B-parameter checkpoints (Base, Chat, LongChat) derived from Orion-14B (a model with LayerNorm and non-standard attention) converted to the Llama-family template. These models load with trust_remote_code=False in the Hugging Face Transformers library.
Theoretical Proof for OPM: A closed-form solution demonstrating that optimal RMSNorm weights replacing LayerNorm under L2 output alignment are equivalent to the original LayerNorm weights in the near-zero-mean regime, rendering the normalization layer conversion training-free.
Efficient Conversion: The method recovers source model behavior using only ~123M tokens on a single A100 GPU over four days, a fraction of the compute required for uptraining.
Zero-Shot Capability Preservation: The method preserves capabilities entirely absent from the transfer corpus, specifically Python programming and 200K-token context handling, demonstrating that KEPT transfers the teacher's inherent knowledge rather than merely relearning the transfer corpus.

Results

The authors evaluated Llamion against the original Orion-14B and other baselines on several benchmarks:

H6 Benchmark (English Few-Shot): Llamion-Base achieved an average score of 60.53 (using the logit objective), recovering nearly all of Orion's performance (60.83). In contrast, parameter mapping alone (OPM) dropped performance to 31.19, and uptraining (UT) without distillation reached 57.77. The logit-based XKD outperformed uptraining by ~2.76 points on the same compute budget.
MT-Bench (Multi-turn Dialogue): Llamion-Chat achieved a score of 7.034, matching Orion-Chat (7.116) within 0.1 points. Uptraining alone (5.709) and OPM alone (1.226) performed significantly worse, highlighting the necessity of the distillation objective for generative coherence.
KoMMLU (Korean MMLU): Llamion-Base scored 66.87%, exceeding the next-best entry on the Open Ko LLM Leaderboard by >7.0 absolute points. This confirms the preservation of Orion's strong Korean language capabilities.
Zero-Shot Transfer:
- Programming: Llamion-Chat generated competent Python code despite the transfer corpus (MIRACL) containing essentially no programming text.
- Long Context: Llamion-LongChat maintained low perplexity on 200K-token sequences, despite the transfer corpus being capped at 4K tokens. This was attributed to the preservation of the RoPE base and the behavior-alignment nature of XKD.

Significance and Claims

The paper claims that KEPT successfully decouples architectural choice from pretraining data. By aligning a model to a frozen teacher rather than a data distribution, the method allows practitioners to adopt deployment-friendly architectures (like the Llama template) without paying the "pretraining tax" or losing capabilities present in the source model.

The authors position this as a solution to the "knowledge loss" problem in architectural transformation. They emphasize that the preservation of unseen-domain capabilities (programming, long context) serves as direct evidence that the method transfers the teacher's learned knowledge rather than re-learning from the transfer corpus. The release of Llamion provides a practical, high-performing, Korean-competent model that is compatible with standard inference stacks, addressing a specific gap in the Open Ko LLM ecosystem.

The authors remain modest regarding the scope, noting that the empirical claims are based on a single source/target pair (Orion-14B to Llama) and that generalization to other pairs (e.g., Gemma to Qwen) has not yet been demonstrated. They also note that OPM is currently specific to the LayerNorm-to-RMSNorm transition.