💬 NLP

LLM generation novelty through the lens of semantic similarity

This paper proposes a scalable, three-stage semantic retrieval framework to measure LLM generation novelty against full pretraining corpora, revealing that models utilize data over longer sequences than previously thought, exhibit domain-specific novelty patterns, and increase novelty through instruction tuning.

Original authors: Philipp Davydov, Ameya Prabhu, Matthias Bethge, Elisa Nguyen, Seong Joon Oh

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Philipp Davydov, Ameya Prabhu, Matthias Bethge, Elisa Nguyen, Seong Joon Oh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to figure out if a student is reproducing content on a test. If the student copies a sentence word-for-word from the textbook, it's obvious reproduction. But what if they read the whole chapter, understood the concept, and then wrote a completely new sentence that means the exact same thing? Is that reproduction, or is that true learning? This is the big question facing scientists who study Artificial Intelligence (AI). These AI models, often called Large Language Models (LLMs), are like super-powered students that have read almost the entire internet. They can write stories, solve math problems, and chat with us. But sometimes, they just repeat what they've read before, and other times, they seem to come up with something new. The tricky part is figuring out the difference between "copying" and "creating" when the AI is so good at hiding its sources. Scientists want to know: Is the AI actually thinking and generalizing, or is it just a fancy parrot repeating patterns it memorized?

This paper introduces a clever new way to solve that mystery. Instead of just looking for exact word matches (like a spell-checker), the authors built a system that looks at the meaning behind the words. They call this "semantic similarity." Think of it like comparing two paintings. If you only look at the specific brushstrokes, two paintings might look totally different. But if you look at the overall image, you might realize they are both painting the same sunset. The researchers used this idea to scan the massive libraries of data the AI was trained on. They didn't just ask, "Did the AI copy this sentence?" They asked, "Did the AI copy the idea of this sentence, even if it used different words?"

To test this, the team used a family of AI models called SmolLM. They set up a three-step investigation. First, they used a fast search tool to find the most similar ideas in the training library. Second, they re-ranked those results to make sure they were looking at the right kind of similarity, not just short, easy matches. Finally, and this is the most important part, they compared the AI's answers to answers written by real humans. This acted like a "calibration" tool, similar to how a chef tastes a dish against a standard recipe to see if the seasoning is right. By doing this, they could measure how "novel" (new and different) the AI's output really was, regardless of whether it sounded fancy or simple.

The results were surprising and changed how we think about these AI models. First, they found that the AI models are drawing on their training data over much longer stretches of text than anyone previously thought. It's not just copying a single sentence; they are weaving together ideas from long paragraphs. Second, the type of task matters a lot. When the AI is asked to do something very strict, like solving a math problem, it tends to be less novel because there's only one right way to say the answer. But when asked to write a story or answer a tricky question, it becomes much more creative and less like a copycat.

Perhaps the most interesting finding was about "instruction tuning." This is when you teach an AI to follow specific commands, like "act like a helpful assistant." The researchers found that this training doesn't just change the AI's style; it actually makes the AI more novel. The instruction-tuned models started using different phrasing and structures that were less similar to their original training data. It's as if teaching the AI to follow instructions gave it the confidence to stop repeating the textbook and start writing its own story.

The authors also discovered that smaller AI models were often more "novel" than the giant ones. The bigger models seemed to stick closer to their training data, almost like they were more afraid to stray from what they had memorized. While the study doesn't prove that these models are "thinking" in the human sense, it strongly suggests that instruction tuning helps them move beyond simple memorization. The researchers released all their tools and data so others can check their work, hoping this new way of measuring "newness" will help us understand exactly what these digital brains are learning and when they are truly creating something new.

Technical Summary: LLM Generation Novelty Through the Lens of Semantic Similarity

Problem Statement

Assessing the generation novelty of Large Language Models (LLMs) is critical for understanding their ability to generalize beyond training data. However, existing evaluation methods face significant limitations:

Lexical Bias: Traditional metrics rely on strict textual overlap (e.g., n-gram non-overlap), which fails to detect paraphrasing, reformatting, or stylistic variations, falsely labeling semantically reused content as novel.
Scale Constraints: Analyzing novelty against full pretraining corpora is computationally prohibitive, leading many studies to rely on curated, domain-specific reference sets that ignore the broader training distribution.
Conceptual Ambiguity: There is no unified definition of novelty that distinguishes between "reproduction" (semantically identical to training data) and "composition" (internally coherent but semantically distinct).

Methodology

The authors propose a three-stage framework to measure semantic novelty at the scale of pretraining corpora. This approach treats novelty as a semantic retrieval problem, leveraging efficient information retrieval pipelines to compare model outputs against their entire pretraining corpus.

1. Conceptual Definition

Semantic novelty is defined as the minimum cosine distance between the embedding of an LLM generation ( $y$ ) and the embeddings of documents in the pretraining corpus ( $C$ ):
$\text{Novelty}(y) = \min_{d \in C} (1 - \cos(\phi(y), \phi(d)))$
where $\phi$ is an embedding function. This definition prioritizes content over surface form, distinguishing between reproduction and composition.

2. The Three-Stage Framework

The framework operates on a dataset scale and is model-agnostic:

Stage 1: Local Candidate Pool Identification (Retrieval):
For each generation, a coarse-grained ranker ( $R$ ) retrieves the top- $n$ passages from the pretraining corpus with the highest potential semantic overlap. This acts as a high-recall filter to create a computationally efficient search space. The authors use GIST embeddings indexed via FAISS for this stage.
Stage 2: Multi-Scale Semantic Re-ranking:
To distinguish between short-range lexical reuse and long-range compositional novelty, the generation and its candidate pool are decomposed into chunks of varying sizes ( $k$ ). A high-precision re-ranker ( $S$ ), specifically ColBERTv2, computes the maximum similarity between each chunk and its respective pool. This stage ensures robustness to paraphrasing and stylistic shifts.
Stage 3: Calibrated Distributional Normalization:
Raw similarity scores are non-linear and sensitive to artifacts like sequence length and domain density. To mitigate this, the framework calibrates model scores against a human novelty baseline ( $H$ $H$ ).
- A "semantic noise floor" ( $\tilde{\mu}^{(k)}_H$ ) is calculated as the median similarity of held-out, human-written text (guaranteed to be absent from the training corpus) for a specific chunk size.
- The final Calibrated Similarity Score is the ratio of the model's raw score to this human baseline:
  $\text{Calibrated Score}(k) = \frac{\text{Raw Similarity Score}}{\tilde{\mu}^{(k)}_H}$
- A score of $\approx 1$ indicates the model's similarity to the training corpus is comparable to the similarity of human text to the corpus (i.e., comparable novelty). Crucially, higher scores indicate higher similarity to the training data, which corresponds to lower novelty. Conversely, lower scores indicate greater deviation from the training data and thus higher novelty.

Experimental Setup

The framework was applied to the SmolLM and SmolLM2 model families (360M and 1.7B parameters, both Base and Instruction-tuned variants).

Corpus: The full pretraining corpora for these models were indexed (~20 TB of embeddings).
Baselines: Human text from the Dolma dataset (not part of the training sets) served as the novelty reference.
Tasks:
- Open-ended generation: Prompted (with context) and unprompted (without context).
- Domain-specific benchmarks: GSM8K (Math), TruthfulQA (Logical), and OpenRewriteEval (Rewriting), filtered strictly for correctness to ensure novelty signals were not driven by hallucinations.

Key Findings

The analysis yielded three primary insights:

Long-Range Reuse Patterns:
Models draw on pretraining data across much longer sequences than previously reported by lexical methods. While n-gram overlap drops to zero for paraphrased content, semantic similarity reveals that models often reproduce concepts found in the training data over hundreds of tokens.
Task-Dependent Novelty:
Novelty varies systematically by domain.
- Constrained Tasks (e.g., GSM8K): Valid solutions have a narrow semantic space, leading to higher similarity scores (lower novelty) as there are limited ways to correctly articulate a proof.
- Open Tasks (e.g., TruthfulQA): Allow for greater semantic variance, resulting in lower similarity scores (higher novelty).
Impact of Instruction Tuning:
Instruction tuning acts as a "novelty catalyst."
- Instruction-tuned models consistently exhibit lower similarity scores (higher novelty) than their base counterparts across all domains.
- This increase is not merely stylistic; even when filtering for correctness, instruction-tuned models produce answers that are compositionally distinct from the raw pretraining data.
- Scale Effect: Smaller models (360M) were found to be more novel than larger models (1.7B) in certain contexts, suggesting larger models may have higher "memorization capacity" in semantic space.

Significance and Claims

The paper claims to provide a scalable, interpretable, and robust method for analyzing LLM generalization.

Beyond Memorization: Unlike membership inference attacks or memorization detection which focus on verbatim reproduction, this framework measures the degree of semantic reuse, offering a continuous spectrum of generalization.
Robustness to Style: By operating in semantic representation space, the method is insensitive to stylistic variations introduced by instruction tuning, which often confound surface-level metrics.
Reproducibility: The authors release ~20 TB of corpus chunks and FAISS indices to enable further research into pretraining-scale novelty.

The authors conclude that semantic novelty analysis is a prerequisite for interpreting what LLMs actually learn, highlighting that instruction tuning shapes not just style, but the fundamental compositional behavior of the model. They explicitly state that their novelty measures are relative and distributional indicators, not tools for identifying specific memorized content or making legal claims about data provenance.