💬 NLP

Memorization Dynamics of Fill-in-the-Middle Pretraining

This paper investigates the memorization dynamics of Fill-in-the-Middle (FIM) pretraining compared to standard left-to-right objectives, revealing that while FIM excels at recovering short or partial spans, it remains heavily dependent on prefix context for verbatim recall and exhibits linear growth in memorization with data repetition.

Original authors: Tobias von Arx, Tanguy Dieudonné

Published 2026-05-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Tobias von Arx, Tanguy Dieudonné

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a student to memorize a long storybook. You want to see how well they can recite specific parts of the story word-for-word. Usually, you teach them by reading the story from start to finish, one word after another. This is called Left-to-Right (LTR) training.

But there's another way to teach them, often used for coding or filling in blanks. You give them the beginning of a paragraph and the end of a paragraph, and ask them to guess what goes in the middle. This is called Fill-in-the-Middle (FIM) training.

This paper asks a simple question: Does teaching a student to "fill in the middle" make them memorize the book differently than teaching them to read straight through?

The researchers set up a controlled experiment. They took two identical AI models (like twins) and taught them the exact same book excerpts. One twin learned the "straight reading" way (LTR), and the other learned the "fill-in-the-blank" way (FIM). They repeated these specific book excerpts anywhere from 1 to 128 times to see how repetition affects memory.

Here is what they found, explained with some everyday analogies:

1. The "Heavy Tail" vs. The "Steady Climber"

When the researchers asked the models to recite a chunk of text starting only from the beginning (the prefix):

The LTR Twin (Straight Reader): This twin is like a person who memorizes a speech perfectly if they hear it enough times. Once they cross a certain threshold of repetition, they can recite long, exact sentences with very high confidence. They have a "heavy tail" of memory—they are great at long, perfect recitations.
The FIM Twin (Fill-in-the-Blank): This twin is a bit different. They are less likely to recite a whole long sentence perfectly. Instead, they are better at remembering shorter snippets or partial matches. Their memory grows more steadily and linearly as they hear the text more often, but they rarely hit that "perfect long recitation" peak as easily as the straight reader.

Analogy: Imagine trying to remember a song.

The LTR student is great at singing the whole chorus perfectly once they've heard it enough, but if you ask them to start from the middle, they might get stuck.
The FIM student is good at humming a few bars or recognizing a melody if you give them a hint from the start or end, but they might not nail the entire 30-second chorus as perfectly as the LTR student.

2. The "Anchor" of the Prefix

The researchers then tested the FIM model in its natural environment: giving it both the start (prefix) and the end (suffix) of a sentence and asking it to fill in the middle.

They discovered something surprising: The end of the sentence (suffix) doesn't help much.

Even though the FIM model sees the end of the sentence, it relies almost entirely on the beginning of the sentence (prefix) to remember the middle part.
If you swap the real beginning of the sentence with a random, unrelated sentence, the model forgets the middle almost instantly.
If you swap the end of the sentence with a random one, the model still remembers the middle quite well, as long as the beginning is correct.

Analogy: Think of the FIM model like a detective trying to solve a crime.

The Prefix is the crime scene photo.
The Suffix is a witness statement about what happened after the crime.
The study found that the detective (the model) solves the case (remembers the text) almost entirely based on the crime scene photo. The witness statement (suffix) is helpful, but if you take away the photo, the detective is lost, even if the witness is still talking.

3. Why the Difference?

Why does the "Fill-in-the-Middle" student behave this way?

LTR Training: Every time the student hears the story, it's the same view: Start $\rightarrow$ Middle $\rightarrow$ End. This reinforces one specific path, making the memory of the whole chain very strong.
FIM Training: Every time the student hears the story, the "middle" part is chopped up differently. Sometimes the middle is the first 10 words, sometimes the next 10. This spreads the memory out. The student learns to recognize pieces of the story from many different angles, but they don't lock onto one single, long, perfect chain as tightly.

The Bottom Line

The paper concludes that how you train a model changes how it memorizes.

If you want a model that can recite long, exact passages of text, Left-to-Right training is stronger.
If you use Fill-in-the-Middle training, the model becomes better at remembering short pieces or partial matches, but it remains heavily dependent on the beginning of the text to trigger that memory.

The researchers also noted that if you only test one specific way of asking the model (like only testing long sentences or only testing short snippets), you might miss these important differences. It's like judging a fish's swimming ability only by how well it jumps out of the water; you need to test it in different ways to understand its true nature.

Technical Summary: Memorization Dynamics of Fill-in-the-Middle Pretraining

Problem Statement
Large language models (LLMs) are known to reproduce training data verbatim, including private information and copyrighted text. While repetition is a primary predictor of memorization, the specific impact of pretraining objectives on this phenomenon remains underexplored. Standard causal language models use a left-to-right (LTR) objective, predicting tokens based solely on preceding context. In contrast, Fill-in-the-Middle (FIM) is a widely used objective for infilling capabilities (e.g., in Code Llama, StarCoder), where a target middle span is predicted using both prefix and suffix contexts separated by sentinel tokens. This paper investigates how the FIM objective alters the dynamics of verbatim memorization compared to standard LTR training, specifically asking whether FIM changes the probability distribution of extracted spans, the role of context (prefix vs. suffix), and the dependence on repetition counts.

Methodology
The authors conduct a controlled study comparing matched Llama 3.2 models (3B and 1B parameter scales) trained on identical data sources and architectures but with different pretraining objectives.

Data Construction: The training corpus combines FineWeb (100B tokens) and a controlled subset of Project Gutenberg excerpts. To isolate memorization effects, Gutenberg excerpts were filtered to remove pre-memorized or outlier windows and then split into 12 "repetition buckets" ranging from 1 to 128 exposures.
Training Regimes: Two corpora were constructed from the same data. The LTR corpus maintained standard autoregressive order. The FIM corpus rewrote examples into a prefix–suffix–middle format using sentinel tokens, with random split points for repeated FIM copies to ensure document-level exposure rather than fixed-span exposure. The models were trained for one epoch (~103B tokens) with a 95%/5% FineWeb/Gutenberg split.
Evaluation:
- Downstream Performance: Evaluated on 8 tasks from the LM Evaluation Harness to ensure memorization differences were not due to general capability gaps.
- Prefix-Only Probing: Models were queried with 100 prefix tokens to predict a 32-token target span. Metrics included exact extraction probability ( $p_z$ ) and ROUGE-L scores.
- Native FIM Probing: Models were queried in their native FIM format, varying the ratio of prefix to suffix context (e.g., 100/0, 50/50, 0/100) to determine the contribution of each side to memorization.
- Attention Analysis: Attention weights were analyzed to understand how models allocate focus between prefix, suffix, and previously generated target tokens.

Key Contributions and Results
The study reveals distinct memorization profiles for FIM and LTR models:

Divergent Extraction Profiles:
- LTR Models: Exhibit a "heavy-tailed" distribution of probabilities. They are more likely to assign high confidence to long, exact continuations. Consequently, LTR models achieve higher rates of strict exact extraction ( $p_z \ge 0.1\%$ ) across the board.
- FIM Models: Distribute probability mass more broadly across partial reconstructions. While they extract fewer windows under strict exact criteria, they demonstrate stronger high-overlap recovery (ROUGE-L) and higher top-k support rates, particularly for shorter spans or at high repetition counts.
- Repetition Dynamics: LTR extraction shows diminishing returns (logarithmic growth) as repetitions increase. FIM extraction grows more linearly with repetitions. However, the span length required for FIM to surpass LTR in extraction rates increases with the length of the target span; longer spans favor LTR's heavy-tailed distribution.
Context Dependency in Native FIM:
- Contrary to the intuition that FIM utilizes both sides equally, prefix context is the dominant driver of memorization in FIM models.
- Native FIM probing shows that memorization support improves monotonically as the context budget shifts from suffix to prefix.
- Replacing the true prefix with a distractor span nearly suppresses memorization, whereas replacing the suffix has a significantly smaller effect. This indicates that even in FIM training, verbatim recall remains strongly anchored in the prefix context, similar to LTR models.
Attention Mechanisms:
- FIM models allocate more attention to the prefix and less to already-seen target tokens compared to LTR models.
- When prefix context is scarce, FIM models compensate by attending more heavily to the preceding tokens of the target span, but this does not fully recover the memorization strength provided by a true prefix.

Significance and Claims
The paper claims that the pretraining objective fundamentally shapes how memorization accumulates. While FIM is designed to improve infilling utility, it does not simply "add" suffix context to LTR memorization; rather, it creates a different memorization profile where:

Memorization is slower to accumulate initially but can exceed LTR performance on short-span extraction at high repetition counts.
The "heavy-tailed" nature of LTR memorization (favoring long exact matches) is replaced by a broader distribution in FIM (favoring partial matches).
Crucially, the study demonstrates that evaluating memorization using only one span length or one probing format (e.g., prefix-only) can miss important nuances. A model might appear less prone to memorization under strict exact extraction but highly prone to high-overlap recovery under FIM probing.

The authors conclude that for models with infilling capabilities, memorization risks are not uniform; they are anchored in the prefix context but manifest differently depending on the extraction geometry and repetition history. This suggests that data curation and memorization audits for infilling-capable models must account for these specific dynamics rather than relying solely on LTR-based metrics.

Limitations
The authors note that their study is limited to frontier-scale models (3B and 1B parameters) and does not extrapolate to larger frontier models. Repetition counts are capped at 128, preventing analysis of asymptotic limits. Furthermore, due to the random decomposition of FIM examples during training, the study cannot trace exact exposures for specific probed spans, limiting the ability to perform fine-grained attribution of specific training instances.

1. The "Heavy Tail" vs. The "Steady Climber"

2. The "Anchor" of the Prefix

3. Why the Difference?

The Bottom Line

Technical Summary: Memorization Dynamics of Fill-in-the-Middle Pretraining

More like this