💬 NLP

Sequence Repetition Enhances Token Embeddings and Improves Sequence Labeling with Decoder-only Language Models

This paper demonstrates that sequence repetition, a less invasive alternative to causal mask removal, effectively enables decoder-only language models to leverage bidirectional context for sequence labeling tasks, yielding superior token embeddings and performance compared to both encoder-only models and unmasked decoders while remaining efficient through the use of intermediate layer representations.

Original authors: Matija Luka Kukić, Marko Čuljak, David Dukić, Martin Tutek, Jan Šnajder

Published 2026-01-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Matija Luka Kukić, Marko Čuljak, David Dukić, Martin Tutek, Jan Šnajder

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "One-Way Street" vs. The "Two-Way Street"

Imagine you are trying to understand a sentence.

Encoder-only models (like BERT) are like a person reading a book who can look both forward and backward. They can see the word before and the word after to understand the meaning of a specific word. This is great for tasks where you need to label every single word (like finding names of people or places).
Decoder-only models (like the ones powering chatbots) are like a person reading a book who is only allowed to look forward. They can only see what comes after the current word. They are built to predict the next word, not to analyze the whole sentence at once.

Because of this "one-way street" limitation, decoder-only models have historically been worse at labeling every word in a sentence compared to the "two-way street" models.

The Old Solution: Breaking the Rules

To fix this, researchers tried a "sledgehammer" approach: they physically removed the rule that stops the model from looking backward. They took the "one-way street" and turned it into a "two-way street" by changing the model's internal architecture.

The downside: This is like rebuilding the engine of a car just to make it drive in reverse. It's complicated, invasive, and requires a lot of re-engineering.

The New Solution: "Sequence Repetition" (The Echo Chamber)

The authors of this paper found a much simpler, "no-tools-required" trick. Instead of changing the model's engine, they just repeated the sentence before feeding it to the model.

The Analogy:
Imagine you are trying to understand the word "Bank" in the sentence: "I went to the river bank."

Normal Decoder: It sees "I went to the river..." and has to guess "bank" based only on what came before. It might think of a money bank.
Sequence Repetition: You feed the model this: "I went to the river bank. I went to the river bank. I went to the river bank."

When the model is processing the second or third copy of the sentence, it has already "seen" the word "river" in the previous copy. Even though the model is still technically only allowed to look "forward," the repetition tricks it into seeing the whole context. It's like having an echo that lets you hear the beginning of the sentence while you are reading the end.

What They Found

1. More Repetitions = Better Understanding
Previous studies suggested that repeating a sentence once was enough, and doing it more didn't help. The authors found the opposite for word-level tasks.

The Finding: Repeating the sentence 2, 4, or even 8 times actually made the model smarter at labeling words.
Why? Each repetition gives the model another "chance" to process the information. It's like reading a difficult paragraph three times; the third time, you understand the nuance much better.

2. It Beats the "Two-Way Street" Models
Surprisingly, using this repetition trick on decoder-only models made them perform better than the traditional "two-way street" models (encoders) and even better than the models where they manually removed the backward-looking rule.

The Result: The simple trick of repeating the text worked better than the complex architectural changes.

3. The "Early Exit" Trick (Saving Time and Money)
Repeating the text makes the input longer, which slows the computer down and costs more money to run.

The Fix: The authors discovered that the model doesn't need to finish reading the entire repeated text to get a good answer. They found that stopping the model halfway through its layers (an "early exit") still gave them high-quality results.
The Analogy: It's like reading a book to understand a plot point. You don't always need to read the last page; sometimes, reading up to the middle of the book is enough to get the gist, and it saves you time.
The Benefit: They could get the same high performance as the full model but compute it 1.4 times faster (or even 4 times faster in some cases) by stopping early.

Summary of the Takeaway

The paper proves that you don't need to rebuild a decoder-only language model to make it good at labeling words. You just need to repeat the input text.

Simple: No code changes to the model's brain.
Effective: It creates a "fake" two-way street that works better than the real thing.
Efficient: By stopping the model early, you can keep it fast and cheap.

The authors conclude that this simple method of "repetition" is a powerful, practical tool that makes modern AI models more versatile without needing complex engineering.

Technical Summary: Sequence Repetition Enhances Token Embeddings and Improves Sequence Labeling with Decoder-only Language Models

Problem Statement
Modern decoder-only Large Language Models (LLMs) are trained autoregressively, conditioned only on the prefix of a sequence via a causal mask. This unidirectional architecture contrasts with Sequence Labeling (SL) tasks (e.g., Named Entity Recognition, Slot Labeling), which inherently benefit from bidirectional context to assign labels to individual tokens. Historically, SL has relied on encoder-only models (e.g., BERT, RoBERTa) that natively support bidirectional attention. While decoder-only models have shown superior scaling capabilities, adapting them to SL typically requires invasive architectural changes, such as removing the causal mask (CM) entirely or selectively. These modifications alter the base model's functionality and require significant re-engineering. Furthermore, while input sequence repetition (SR) has been explored for improving sequence-level representations, its efficacy for token-level embeddings and SL performance remains underexplored.

Methodology
The authors propose Sequence Repetition (SR) as a non-invasive alternative to enable bidirectionality in decoder-only models without modifying their core architecture.

Mechanism: The input sequence $s$ is repeated $r$ times to form a new sequence $s_k$ of length $k \cdot n$ (where $k = r+1$ ). Due to the causal mask, tokens in later repetitions can attend to all tokens in previous repetitions. Theoretical analysis of the attention weight matrix shows that as $k$ increases, the proportion of fully bidirectional attention blocks approaches 1, effectively allowing the model to process tokens with full context.
Experimental Setup: The study evaluates SR across seven decoder-only models (Gemma-7B, Gemma2-2B/9B, Qwen3-1.7B/4B/8B, Mistral-7B) and four SL datasets (CoNLL03, SemEval-2014 Rest14, NLU++, ACE05).
Baselines: Performance is compared against:
1. Masked Decoder: Standard fine-tuning with no repetitions ( $r=0$ ).
2. Full Unmasking: Removing the causal mask entirely.
3. Middle Unmasking: Removing the causal mask only from the middle $N/3$ layers (a parameter-efficient adaptation strategy).
4. Encoder-only Models: RoBERTa and ModernBERT.
Efficiency Strategy: To mitigate the computational cost of increased sequence length, the authors employ Early Exiting, extracting embeddings from intermediate layers ( $L \in \{9, 14, 19, 24\}$ ) rather than the final layer.

Key Contributions

Effectiveness of SR for Token-Level Tasks: The paper demonstrates that SR is a simple, practical strategy that inherently enables bidirectionality in decoder-only models. SR yields token-level embeddings that outperform both strong encoder-only models and unmasked decoders across multiple SL benchmarks.
Benefit of Multiple Repetitions: Contrary to prior findings on sequence-level tasks where additional repetitions offered diminishing returns or degradation, this work finds that increasing the number of repetitions ( $r > 1$ ) consistently improves SL performance. The authors hypothesize that additional repetitions increase the model's effective processing capacity for each token.
Efficiency via Early Exiting: The study shows that extracting embeddings from intermediate layers (specifically layers 19 and 24 in a 32-layer model) combined with SR achieves performance comparable to final-layer embeddings while significantly reducing computational overhead (up to $1.39\times$ more efficient).

Results

Performance Gains: SR models consistently achieve the highest Micro F1 scores. For instance, Mistral-7B with SR ( $r=4$ ) achieved an average score of 83.79%, outperforming Middle Unmasking (83.35%) and Full Unmasking (83.34%). SR also surpassed encoder-only baselines (RoBERTa: 78.53%, ModernBERT: 76.08%).
Repetition Count: Performance generally improves as $r$ increases from 0 to 4 or 8. While smaller models (e.g., Gemma2-2B) showed performance saturation or slight drops after $r=2$ or $r=4$ , larger models (e.g., Mistral-7B, Qwen3-8B) continued to benefit from higher repetition counts.
Stability: SR not only improved mean performance but also reduced the variance (standard deviation) of results compared to unmasking strategies.
Early Exiting: Embeddings from layer $L=14$ or higher matched or exceeded the performance of the final layer ( $L=32$ ) in several cases, validating the use of intermediate layers for efficiency without sacrificing quality.
Model Size: Even smaller decoder-only models (e.g., Qwen3-0.34B via early exiting) trained with SR performed on par with or better than the 0.35B parameter RoBERTa, suggesting that SR allows smaller decoders to compete with specialized encoders.

Significance and Claims
The paper claims that Sequence Repetition alleviates the structural limitations of decoder-only models, enabling them to perform effectively on inherently bidirectional token-level tasks without architectural modification. The authors position SR as a more efficient and adaptable alternative to causal mask removal, broadening the applicability of decoder-only LMs to tasks previously dominated by encoders. The findings underscore that intermediate layers are critical for downstream tasks and that increasing repetition counts can enhance processing capacity rather than degrade performance, challenging previous assumptions about repetition in LMs. The work suggests that SR offers a favorable quality-efficiency trade-off, particularly when combined with early exiting strategies.