💬 NLP

Where Does Authorship Signal Emerge in Encoder-Based Language Models?

This paper demonstrates that while stylistic authorship features are available at all layers of encoder-based models, the choice of scoring mechanism (mean pooling vs. late interaction) causally determines the specific layer where the model consolidates these signals, thereby explaining significant performance differences between otherwise identical models.

Original authors: Francis Kulumba, Guillaume Vimont, Laurent Romary, Florian Cafiero

Published 2026-05-20

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Francis Kulumba, Guillaume Vimont, Laurent Romary, Florian Cafiero

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a team of expert detectives (the encoder) who are very good at reading books and noticing subtle details like sentence length, punctuation habits, and favorite words. These detectives can spot who wrote a text just by looking at these clues.

Now, imagine you have two different ways to ask these detectives for their final verdict:

The "Summary" Method (Mean Pooling): You ask the detectives to write a single, short summary sentence that captures the entire book. Then, you compare these summary sentences to decide if the books were written by the same person.
The "Spot-Check" Method (Late Interaction): Instead of asking for a summary, you ask the detectives to look at specific sentences or words in Book A and find the most similar sentence or word in Book B. You let them compare the details directly, piece by piece.

The Big Surprise
The paper reveals a shocking fact: Even though the detectives (the AI model) are exactly the same, and they are trained on the exact same books, the "Spot-Check" method is four times better at solving the mystery than the "Summary" method.

Why? The authors wanted to know: Is the Summary method making the detectives "dumber" or "forgetful" during training?

The Investigation: What the Paper Found

1. The Clues Are Always There (Feature Availability)
The researchers used a special tool (like a magnifying glass) to check if the detectives actually saw the clues (word length, punctuation, etc.) at every stage of their thinking process.

The Result: They found that both methods saw the clues perfectly. The "Summary" detectives weren't missing anything. The clues were available in the first layer, the middle layer, and the last layer. The problem wasn't that the detectives failed to learn the clues; it was that the "Summary" method forced them to throw away the details too early.

2. The Bottleneck: When to Decide? (Consolidation)
This is the core discovery. The way you ask for the answer changes when the detective has to make up their mind.

The "Summary" Method: Because you need a single sentence to summarize the whole book, the detective is forced to compress all the information into that one sentence very early in the process (around the middle of the book). They have to stop looking at the fine details and just focus on the "big picture" too soon. It's like trying to describe a complex painting by only looking at the center of it and ignoring the edges.
The "Spot-Check" Method: Because you can compare specific words directly, the detective doesn't need to compress everything early. They can keep looking at the fine details all the way to the very end of the book. They only make their final decision at the last possible moment, using the most refined, detailed information available.

3. The Training Journey
The paper also watched how the detectives learned over time:

The "Summary" detectives learned from the top down. They started by trying to get the "big picture" right immediately, then slowly tried to fix the details in the middle layers.
The "Spot-Check" detectives took a different path. At first, they tried to cheat by matching simple, shallow words (like "the" or "and"). But as the training got harder, they realized that wasn't enough. They learned to ignore those easy shortcuts and started digging deep into the complex, abstract layers of the text to find the real author's style.

The Simple Analogy
Think of it like packing a suitcase for a trip:

Mean Pooling is like being told, "You can only fit everything into one small box." You have to crush your clothes and throw away the shoes immediately because you know the box is small. You lose a lot of information.
Late Interaction is like being told, "You can pack a huge suitcase, but you have to show me exactly which socks match which shirt when you get to the airport." You can keep all your clothes separate and organized until the very end. You don't lose anything.

The Bottom Line
The paper concludes that the "Summary" method isn't failing because the AI is bad at learning. It's failing because the rules of the game (how we ask for the answer) force the AI to throw away its best clues too early. The "Spot-Check" method wins because it lets the AI keep all its detailed clues right up until the final moment of judgment.

Technical Summary: Where Does Authorship Signal Emerge in Encoder-Based Language Models?

Problem Statement

Authorship attribution (AA) systems typically employ a contrastive learning paradigm where a pretrained text encoder generates representations for passages, and a scoring function compares these representations to determine authorship similarity. Recent work (Kulumba et al., 2025) revealed a striking puzzle: models fine-tuned with the same pretrained encoder, training data, and contrastive loss can exhibit a four-fold performance gap solely based on their scoring mechanism. Specifically, models using mean pooling (averaging token embeddings into a single vector) significantly underperform compared to those using Late Interaction (LI) (comparing token vectors directly via MaxSim).

The central question addressed by this paper is the source of this discrepancy. Two hypotheses were considered:

Representation Learning Hypothesis: Different scoring mechanisms cause the encoder to learn fundamentally different internal representations during fine-tuning.
Readout Hypothesis: The encoder learns similar representations regardless of the scorer, and the performance gap arises from how those representations are read out (scoring) at inference time.

Methodology

The authors employ mechanistic interpretability tools to distinguish between feature availability (whether information is present in the hidden states) and feature use (whether the scoring mechanism causally relies on that information). The study utilizes the following experimental setup and tools:

Models: Three fine-tuned models sharing a ModernBERT-base backbone (23 layers, 149M parameters) were trained on the HALvest-Contrastive corpus (a scholarly corpus where topic is decorrelated from authorship). The models differed only in their scoring mechanism:
- Layerwise (Mean Pooling): Averages token embeddings before scoring.
- Late Interaction (LI): Uses token-level MaxSim.
- Patch-Level Late Interaction (PLI): A hybrid using bigram patches with MaxSim.
- A control model (E5, zero-shot) was included.
Probe Set: A curated set of 148 triplets (anchor, positive, negative) with controlled token lengths (target 130) and stratified by difficulty (Tier A: disjoint authors; Tier B: partial author overlap; Tier C: cross-domain).
Analysis Tools:
1. LISA Probes: Linear regression probes trained at each of the 23 encoder layers to measure the linear readability of stylistic features (e.g., word length, punctuation density, function-word frequency).
2. Residual Stream Patching: A causal intervention where hidden states of a "corrupt" (negative) passage are replaced with those of a "clean" (positive) passage at specific layers. Rank recovery measures whether the model correctly ranks the positive above the negative after this intervention, identifying the layer where authorship signal becomes causally necessary.
3. Score Sensitivity: Measures the absolute change in scoring output when a single layer's activations are restored.
4. Training Dynamics: Patching analysis applied across eight training checkpoints to observe the evolution of signal consolidation.

Key Results

1. Feature Availability is Invariant

LISA probe results (Figure 3) show that stylistic features are linearly readable from the hidden states of all models (Mean Pooling, LI, PLI, and the control E5) at all layers with nearly identical $R^2$ scores.

Implication: The pretrained backbone already encodes stylistic features. Contrastive fine-tuning does not create new stylistic representations, nor do different scorers force the encoder to learn different ones. This rules out the Representation Learning Hypothesis.

2. Causal Patching Reveals a Scoring-Dependent Depth Profile

Rank recovery curves (Figure 4) demonstrate a distinct "consolidation point" where the model's ability to recover the correct ranking depends on the scoring mechanism:

Mean Pooling: Signal consolidation occurs early. Rank recovery crosses the chance baseline at layer 9 and reaches near-perfect recovery by layer 13.
Late Interaction (LI) & PLI: Signal consolidation is deferred. Rank recovery remains below chance until layers 14–16, reaching high recovery only in the final layers.
Gap: There is a consistent 6-layer gap in the depth at which authorship signal becomes causally necessary.

3. Gradient Structure and Information Loss

The paper derives a theoretical explanation based on gradient distribution:

Mean Pooling: Produces a dense, uniform gradient across all tokens. This forces the encoder to compress all authorship-relevant information into a single vector early in the network to survive the averaging process. This creates a "consolidation bottleneck" and leads to information loss (Data Processing Inequality), as permutation-invariant averaging discards positional structure.
Late Interaction: Produces a sparse, selective gradient (only tokens selected by argmax receive updates). This allows the encoder to preserve per-token structure and defer consolidation to deeper, more abstract layers.

4. Training Dynamics

Analysis of training checkpoints (Figure 6) reveals distinct learning trajectories:

Mean Pooling: Learns top-down. The model initially relies on upper layers, and the consolidation point migrates downward (from layer 15 to layer 9) as training progresses, progressively recruiting deeper layers to satisfy the early consolidation bottleneck.
Late Interaction: Shows a transient early-layer spike (layers 5–10) at intermediate training steps, exploiting shallow lexical matches. As training continues and negatives become harder, the model suppresses this shortcut and shifts reliance to deeper layers (19+).
PLI: Exhibits a gradual emergence without the early spike, showing a unique mid-layer hump (layers 10–15) in the final checkpoint.

Significance and Claims

The paper claims to resolve the puzzle of the four-fold performance gap in authorship attribution by demonstrating that the bottleneck is not in what the encoder learns, but in how the scorer reads it out.

Dissociation of Availability and Use: The study provides empirical evidence that high feature availability (probing accuracy) is a poor proxy for task performance when scoring mechanisms differ. All models possess the necessary stylistic features; the difference lies in the causal necessity of those features at specific depths.
Mechanism of Failure: Mean pooling forces early signal consolidation, which destroys fine-grained stylistic information (such as word order and specific token interactions) required for high-performance AA. Late interaction preserves this granularity by allowing the model to utilize deeper, more contextualized representations.
Theoretical Contribution: The work links the gradient structure of the scoring function (dense vs. sparse) directly to the depth of signal consolidation in the encoder, offering a mechanistic explanation for why interaction-based systems consistently outperform pooled representations in AA tasks.

The authors conclude that the primary limitation in contrastive authorship attribution is not the encoder's capacity to encode style, but the scoring mechanism's ability to preserve and exploit that style without premature compression.