💬 NLP

Seq vs Seq: An Open Suite of Paired Encoders and Decoders

The paper introduces the Ettin suite, a collection of paired encoder-only and decoder-only models trained on identical data and recipes that establishes new state-of-the-art performance for their respective sizes while demonstrating that native architectures significantly outperform adapted models on their specialized tasks.

Original authors: Orion Weller, Kathryn Ricci, Marc Marone, Antoine Chaffin, Dawn Lawrie, Benjamin Van Durme

Published 2026-03-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Orion Weller, Kathryn Ricci, Marc Marone, Antoine Chaffin, Dawn Lawrie, Benjamin Van Durme

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are building a library of books. For a long time, the librarians (the AI community) have been obsessed with two specific types of book readers:

The "Decoder" (The Storyteller): This reader is amazing at writing new stories from scratch. If you give them the first sentence of a story, they can write the whole rest of it. They are the stars of the show right now (like the famous Llama or GPT models).
The "Encoder" (The Analyst): This reader is terrible at writing new stories, but they are incredible at understanding what they've already read. If you ask them, "Is this sentence happy or sad?" or "Do these two sentences mean the same thing?", they are lightning fast and very accurate.

The Problem:
For years, everyone stopped building "Analysts" because "Storytellers" became so popular. People started thinking, "Why bother with a specialist Analyst when a Storyteller can do everything?" They tried to teach Storytellers to act like Analysts by giving them extra homework. But it felt like comparing apples to oranges because the Storytellers were huge and trained on different data than the old Analysts. We didn't know if the Storyteller was actually better, or if it just had more training.

The Solution: The "Ettin" Suite
The authors of this paper built a brand new library of models called Ettin (named after a two-headed giant from Norse mythology).

Think of Ettin as a twin experiment. They built pairs of models:

Twin A: A pure Analyst (Encoder).
Twin B: A pure Storyteller (Decoder).

Here is the magic trick: They are twins. They are the exact same size, they ate the exact same food (training data), and they were raised in the exact same house (same training recipe). The only difference is their job description: one is told to "fill in the blanks" (Analyst), and the other is told to "write the next word" (Storyteller).

What They Discovered

Specialists Win:
When they tested the twins, the results were clear.
- If you need to classify something (like sorting emails into "Spam" or "Not Spam") or search for information, the Analyst (Encoder) wins every time.
- If you need to generate creative text or write a story, the Storyteller (Decoder) wins every time.
You Can't Just "Train" Your Way Out of It:
Many people tried to fix this by taking a Storyteller and giving it extra homework to make it act like an Analyst.
- The Analogy: Imagine taking a professional novelist and forcing them to take a speed-reading test. They might get a little better, but they will never beat a professional speed-reader who has been doing it their whole life.
- The paper found that even after training the Storyteller for a massive amount of time, it still couldn't beat the Analyst at analysis tasks. In fact, a small Analyst was often better than a giant Storyteller trying to do the Analyst's job.
The "Bias" Surprise:
The authors also looked at how these twins "think" about gender. They found that the Analysts (Encoders) were more likely to use neutral language (like "they"), while the Storytellers (Decoders) were more likely to lean toward male pronouns. This shows that the way you train a model changes its personality, not just its skills.

Why This Matters
This paper is like a fair referee in a boxing match. Before, people were comparing a heavyweight champion (the big Storyteller) against a lightweight amateur (the old Analyst) and saying, "See? The Storyteller is better!"

Now, with Ettin, we have a heavyweight Storyteller fighting a heavyweight Analyst. The verdict? They are both champions, but in different weight classes.

If you want to build a chatbot or write a novel? Use a Decoder (Storyteller).
If you want to build a search engine, a spam filter, or a medical diagnostic tool? Don't force a Storyteller to do it. Go find a dedicated Analyst (Encoder).

The authors released all their blueprints, data, and twins for free, so other scientists can stop guessing and start building the right tool for the right job.

1. Problem Statement

The Large Language Model (LLM) community has largely shifted toward decoder-only architectures (GPT-style) due to their superior performance in text generation. Conversely, encoder-only architectures (BERT-style) remain dominant for classification and retrieval tasks but have seen limited recent development, often relying on models from 2019 (e.g., BERT, RoBERTa).

Previous attempts to compare these two architectures suffer from a critical flaw: incomparability. Existing studies compare models with different parameter counts, training datasets, learning schedules, and architectural nuances. This makes it impossible to isolate whether performance differences are due to the architecture (encoder vs. decoder) or the training recipe/data. Furthermore, there is a growing trend of adapting decoder models to encoder tasks (and vice versa) via continued pre-training, but the efficacy of this "cross-objective" training remains unclear without a controlled baseline.

2. Methodology

The authors introduce ETTIN (named after the two-headed Norse giant), a suite of paired encoder-only and decoder-only models trained under identical conditions to enable a fair, "apples-to-apples" comparison.

A. Model Suite and Architecture

Scale: The suite includes 10 models (5 pairs) ranging from 17 million to 1 billion parameters.
Configurations: Both encoders and decoders share the exact same hyperparameters (learning rate, weight decay, layers, hidden size) as detailed in Table 1.
- Differences: The only architectural differences are the attention mechanism (bidirectional for encoders, causal for decoders) and the pre-training objective (Masked Language Modeling - MLM for encoders, Causal Language Modeling - CLM for decoders).
Design Philosophy: The models follow a "deep but thin" design inspired by MobileLLM, with specific scaling for the 1B model.

B. Training Data and Recipe

Data Source: To replicate the state-of-the-art (SOTA) ModernBERT recipe using open data, the authors constructed a dataset mix from DCLM and Dolma v1.7 (specifically the newer "Dolmino" filtered sources).
Training Phases: The training follows a three-stage process (Total tokens: ~2T for smaller models, scaled for 1B):
1. Base Pre-training: 1.7T tokens with a wide mix of data (news, code, web crawl) using a trapezoidal learning rate.
2. Mid-training (Context Extension): 250B tokens using higher-quality filtered data (DCLM, math, StackExchange) and extended context lengths (up to 8k tokens).
3. Decay Phase: 50B tokens using high-quality reference data (textbooks, Wikipedia, books) with a decaying learning rate.
Checkpointing: Models are checkpointed every 8.5B tokens, resulting in 236 checkpoints per model, allowing for granular analysis of learning dynamics.

C. Cross-Objective Training Experiments

To test the adaptability of architectures, the authors performed cross-objective continued pre-training:

Decoder $\to$ Encoder: Taking a final decoder model and continuing pre-training with MLM (specifically using MNTP, where the masked token is predicted using the previous hidden state to align with CLM).
Encoder $\to$ Decoder: Taking a final encoder model and continuing pre-training with CLM.

Duration: 50B tokens of cross-objective training (significantly longer than previous works like LLM2Vec).

3. Key Contributions

The ETTIN Suite: The first open-data suite of paired encoder/decoder models trained on the same data, architecture, and recipe across multiple scales (17M–1B).
SOTA Performance: The models achieve state-of-the-art results for their respective sizes among open-data models, beating ModernBERT (encoders) and Llama 3.2/SmolLM2 (decoders) despite using the same training recipe.
Replication of ModernBERT: The paper provides the first open-data replication of the ModernBERT training process, allowing the community to build upon it.
Comprehensive Artifact Release: The authors open-sourced training data, training order segmented by checkpoint, and 200+ checkpoints, enabling future research into learning dynamics and data influence.

4. Key Results

A. Architecture Specialization

Encoders: Excel at classification (GLUE, MNLI) and retrieval (MS MARCO, MTEB).
Decoders: Excel at generative tasks (ARC, HellaSwag, LAMBADA, TriviaQA).
Scaling: Encoders generally outperform decoders of the same size on classification/retrieval, and decoders outperform encoders on generation.

B. Cross-Objective Training Limitations

Ineffectiveness of Adaptation: Continued pre-training a decoder to act as an encoder (or vice versa) does not close the performance gap.
- Example: A native 400M encoder outperforms a 1B decoder that has been continued-trained with MLM on the MNLI task.
- Example: A native 1B decoder significantly outperforms a continued-trained encoder on generative tasks.
Conclusion: Simply adding more tokens to a model with the "wrong" objective is insufficient to overcome the architectural bias. The native architecture remains superior for its specific task.

C. Learning Dynamics & Bias (Case Study)

Gender Bias: Using the WinoGender benchmark, the authors analyzed pronoun prediction.
- Encoders showed a higher tendency to predict gender-neutral pronouns compared to decoders.
- Decoders showed a stronger bias toward male pronouns as model size increased.
- This suggests that the pre-training objective (MLM vs. CLM) influences how models learn social biases, even when trained on identical data.

5. Significance and Implications

Scientific Rigor: This work establishes a new standard for comparing LLM architectures by eliminating confounding variables (data, recipe, scale).
Practical Guidance:
- For classification and retrieval tasks, especially in resource-constrained environments (<1B parameters), training a native encoder is superior to adapting a decoder.
- For generative tasks, native decoders are essential.
- The "cross-objective" adaptation strategy (e.g., turning a massive LLM into an embedding model via fine-tuning) may be suboptimal compared to training a smaller, native encoder from scratch.
Future Research: The release of checkpointed data and training orders allows researchers to study how models learn, the impact of data recency, and the specific effects of training objectives on model behavior (e.g., bias, reasoning).

In summary, the ETTIN suite confirms that while decoder models dominate generation, encoder models remain the optimal choice for understanding tasks, and that architectural specialization cannot be easily overcome by simply changing the training objective on top of a pre-trained model.