💬 NLP

Comparing Transformers and Hybrid Models at the Token Level

This paper analyzes token-level performance differences between pure transformer and hybrid language models using Olmo 3 and Olmo Hybrid, revealing that hybrid models excel at predicting semantic content and tracking state via recurrent layers, while transformers outperform on syntactic bracket matching and $n$ -gram copying tasks.

Original authors: Yanhong Li, William Merrill

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yanhong Li, William Merrill

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach two different types of students how to write the next sentence in a story. One student is a Super-Reader (the Transformer), and the other is a Super-Reader with a Notebook (the Hybrid model).

The paper asks: When does the student with the notebook actually do a better job than the one who just relies on their memory of what they've read so far?

Here is the breakdown of their findings using simple analogies:

1. The Two Students

The Super-Reader (Transformer): This student has an incredible memory for everything they just read. If you say, "The cat sat on the...", they can instantly remember that "mat" was mentioned three sentences ago. They are great at copying patterns and finishing sentences that follow a strict rule (like matching a parenthesis ( with a )).
The Super-Reader with a Notebook (Hybrid): This student also has a good memory, but they also carry a notebook where they write down the "state" of the story as they go. They track who owns what, who is doing what, and how the plot is evolving.

2. When the "Notebook" Student Wins

The paper found that the student with the notebook (the Hybrid) is much better at predicting the next word when the story requires keeping track of a changing situation.

The "Content" Advantage: The Hybrid student shines when predicting open-class words (like nouns, verbs, and adjectives). These are the "meaty" parts of a sentence that carry new meaning.
- Analogy: Imagine a detective story. If the text says, "The detective found a red ...," the Super-Reader might guess "hat" or "car" based on common phrases. But the Hybrid student, having written down in their notebook that the suspect was wearing a red coat earlier, is more likely to guess "coat" because they are tracking the story's state, not just the immediate words.
The "State" Advantage: The Hybrid is better at tasks where you have to remember a role or an object over a long distance.
- Example: "Liam is the violinist. Naomi is the pilot. ... (lots of words) ... The violinist reviewed the report." The Hybrid is better at remembering that "violinist" refers to Liam, even after a long gap, because it updated its internal state.

3. When the "Memory" Student Wins

Surprisingly, the student without the notebook (the pure Transformer) is actually better in specific situations where the answer is already staring them in the face.

The "Closing" Advantage: When the text needs to close a structure, the Transformer wins.
- Analogy: If the text has an open bracket (, the Transformer knows exactly what the closing bracket ) should be. It doesn't need a notebook to remember that ( needs a ); it just looks at the visible text. The Hybrid student sometimes gets confused here, perhaps because it's too busy tracking the "story state" and misses the simple structural rule.
The "Copying" Advantage: If the text is repeating a long phrase (like a copy-paste error or a repetitive list), the Transformer is unbeatable.
- Analogy: If the text says, "The quick brown fox jumps over the lazy dog. The quick brown fox jumps over the...", the Transformer just copies the pattern it sees right in front of it. The Hybrid student tries to figure out the "meaning" of the repetition, which is unnecessary work, so it performs worse.

4. The "Function Word" Mystery

The paper also looked at small, boring words like "the," "is," "and," or "to" (function words).

The Hybrid student is only slightly better at these.
Why? Because these words are like a small, closed list. There are only so many of them. Once you know the category (e.g., "this is a preposition"), there isn't much room for the notebook to help you guess which specific one to use. The advantage of the notebook is biggest when there are thousands of possibilities (like nouns and verbs) and you need context to pick the right one.

5. The Big Takeaway

The paper concludes that Hybrid models aren't just "better at everything." They are better at specific things:

Tracking the plot: Keeping track of who owns what, what variables are set in code, or what the current topic is.
Predicting new ideas: Guessing the next "content word" in a story or code.

However, they are not better at:

Simple copying: Repeating what was just said.
Structural closing: Matching brackets or tags that are already open.

Why This Matters for the Future

The authors suggest that if we want to build even smarter AI, we shouldn't just look at the "average" score. Instead, we should look at which words the AI gets right.

If an AI is bad at "closing brackets," we know it's struggling with structure.
If an AI is bad at "tracking who owns what," we know it's struggling with its internal notebook (state tracking).

By breaking the results down word-by-word, researchers can see exactly what kind of "brain" (memory vs. notebook) an AI is using and fix the specific parts that are weak.

Technical Summary: Comparing Transformers and Hybrid Models at the Token Level

Problem Statement

Hybrid language models, which combine attention mechanisms with recurrent sequence layers, have demonstrated empirical superiority over pure transformers in terms of loss and downstream benchmarks. Theoretically, this advantage is motivated by the complementary expressivity of the two architectures: attention excels at retrieving information from visible prefixes (useful for copying and structural matching), while recurrent layers excel at constructing and updating latent states for ordered state-tracking computations.

However, it remains unclear which specific data types or prediction capabilities drive these empirical gains, and whether the observed improvements align with the theoretical advantages motivating hybrid architectures. This paper addresses the question: Which individual token occurrences account for the lower average loss of a hybrid model relative to a matched transformer, and do these tokens correspond to the theoretical strengths of recurrence versus attention?

Methodology

The authors conduct a fine-grained, token-level analysis comparing two matched 7B-scale models from the same recipe family: Olmo 3 7B (pure transformer) and Olmo Hybrid 7B (interleaved attention and recurrent layers).

Core Metric

For a target token $x_i$ given a prefix $x_{<i}$ , the authors compute the paired token-level loss gap:
$\Delta_i = \ell^{Tr}_i - \ell^{Hyb}_i = \log p^{Hyb}(x_i | x_{<i}) - \log p^{Tr}(x_i | x_{<i})$
A positive $\Delta_i$ indicates the hybrid model assigns a higher probability (lower NLL) to the token than the transformer.

Analysis Framework

The study employs three complementary analytical approaches across prose, code, and markup domains:

Observational Tag-Stratified Analysis:
- Target tokens are aligned with surface tags (e.g., POS tags for prose; identifiers, strings, delimiters for code/markup).
- Raw Summaries: Compute the mean $\Delta_i$ for each tag category to determine absolute hybrid advantage.
- Regression Robustness: Fit linear regressions controlling for confounding variables (difficulty, frequency, position, subword status, and local reuse) to determine if tag-level patterns persist after accounting for these factors.
Controlled Synthetic Probes:
- To isolate specific computational requirements, the authors evaluate models on three synthetic tasks with fixed templates and varying dependency distances ( $d \in \{32, \dots, 1024\}$ $d \in {32, \dots, 1024}$ ):
  - Pronoun Memory: Requires maintaining a role-binding state to select the correct pronoun.
  - Entity Tracking: Requires maintaining entity-attribute bindings to answer queries.
  - Structural Closure: Requires predicting a closing delimiter based on a visible opener.
Filtered Evaluation (Proof of Concept):
- Applied to 1B-scale pretraining runs (Transformer, Hybrid, Pure RNN) to demonstrate how token-level decompositions can sharpen architecture diagnostics during training.

Key Results

1. Non-Uniform Token-Level Gains

The hybrid model's advantage is not uniform across all tokens.

Open-Class Content Words: The hybrid shows the largest loss reduction on open-class content-bearing categories (nouns, verbs, adjectives, adverbs). In prose, the raw gap for content words is approximately 61% larger than for function words.
Closed-Class Function Words: The advantage is smaller and more heterogeneous for closed-class function words (determiners, prepositions, pronouns).
Correlation with Vocabulary Size: There is a positive correlation between the size of a tag's vocabulary (openness) and the hybrid's advantage. Larger vocabularies (open classes) show greater hybrid gains, while small, closed inventories show smaller gains.

2. Structural Delimiters: Openers vs. Closers

Opening Delimiters: The hybrid is significantly favored on opening brackets and tags across all domains (prose, Python, HTML, LaTeX). These tokens often initiate a new scope or state update.
Closing Delimiters: The hybrid advantage nearly vanishes or reverses on closing delimiters. The transformer performs competitively or better here, as closing tokens satisfy structural obligations already established by the visible prefix (a retrieval/closure task).

3. Repeated N-Grams and Copying

The hybrid advantage collapses on repeated n-grams. As the length of the repeated span increases ( $n=1$ to $16$), the raw loss gap shrinks rapidly toward zero.
Regression analysis confirms that repetition features shift the gap toward the transformer. This suggests that when the target is predictable via visible-prefix copying, the attention mechanism is sufficient, and the recurrent state offers no additional benefit.

4. Synthetic Probe Findings

State-Conditioned Tasks (Pronoun Memory, Entity Tracking): The hybrid consistently outperforms the transformer, maintaining accuracy above chance even at long distances where the transformer dips below chance.
Structural Closure: The transformer outperforms the hybrid on closure tasks, regardless of distance.
Conclusion: The critical distinction is not merely dependency length, but the type of computation: state-conditioned readout favors recurrence, while visible-prefix retrieval/closure favors attention.

5. Filtered Evaluation Utility

In 1B-scale training runs, aggregate validation loss showed minimal separation between architectures. However, filtered losses revealed distinct regimes:

State-Oriented Filter (Top-10 Open Class, No Copy): Amplified the gap, showing the Hybrid significantly outperforming the Pure Transformer and Pure RNN.
Copy Filter (Repeated 5-grams): Revealed that Pure RNNs lag significantly behind attention-based models on retrieval tasks, a weakness masked by aggregate loss.

Significance and Claims

The paper claims that the empirical success of hybrid models is driven by specific, theoretically consistent mechanisms rather than a general "better" performance across all tasks.

Validation of Expressivity Theory: The results align with expressivity theory. Recurrent layers improve predictions that leverage the semantic state of a document (e.g., tracking entities, roles, or program variables), while attention mechanisms remain superior for tasks relying on visible-prefix retrieval (copying) or structural closure (matching openers).
Refining State-Tracking Benchmarks: The authors argue that existing closed-world state-tracking benchmarks (like the $A_5$ word problem) are insufficient for distinguishing future recurrent layers because they can be solved by structured linear-RNNs. They propose discourse-state tracking as a more demanding target, where the state grows dynamically with new entities and relations, better reflecting the needs of natural language and code.
Diagnostic Tool for Architecture Search: The paper proposes that filtered token losses serve as a high-signal diagnostic tool during pretraining. By decomposing aggregate loss into "state-conditioned" and "copy/retrieval" components, researchers can identify whether an architectural change improves state representation or retrieval capabilities, rather than relying on a single aggregate perplexity number that may obscure trade-offs.

The authors conclude that token-level decompositions can sharpen pretraining diagnostics, expose capability trade-offs hidden by aggregate metrics, and guide the design of future hybrid architectures.