💬 NLP

Theory-Grounded Evaluation Exposes the Authorship Gap in LLM Personalization

This paper demonstrates that grounding LLM stylistic personalization evaluation in authorship verification theory (specifically using the LUAR metric) reveals a significant "authorship gap" where current methods fail to match human style, a critical failure that remains invisible to uncalibrated, ad hoc metrics like LLM-as-judge or stylometrics which produce inconsistent and uncalibrated results.

Original authors: Yash Ganpat Sawant

Published 2026-04-30

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yash Ganpat Sawant

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a very talented, but generic, robot writer to sound exactly like your favorite famous author. You want the robot to capture that author's unique "voice"—their specific way of using words, their rhythm, and their personality.

This paper is like a detective report that says: "We tried to teach the robot, but we realized we didn't have a real ruler to measure if we succeeded. When we finally built a proper ruler, we discovered the robot is actually failing completely."

Here is the breakdown of the story using simple analogies:

1. The Problem: Measuring with a Tape Measure vs. a Ruler

For a long time, researchers tried to see if robots could mimic human writers. But they were using the wrong tools.

The Old Way (Ad Hoc Metrics): Imagine trying to measure the height of a building by asking a friend, "Does this look tall to you?" or counting how many bricks are on the roof. These methods are vague. They might tell you one building looks "taller" than another, but they can't tell you if it's actually 100 feet tall or just 10 feet tall.
The New Way (Theory-Grounded): The authors brought in a real, calibrated ruler based on decades of "Authorship Science" (the study of how to identify who wrote a text). This ruler has clear markings:
- The Ceiling: How similar two texts by the same human author look (Score: 0.756).
- The Floor: How similar two texts by different humans look (Score: 0.626).

2. The Big Discovery: The "Authorship Gap"

The researchers tested four different ways to make the robot sound like a human. They used the new, real ruler to check the results.

The Shock: Every single robot attempt scored between 0.48 and 0.50.
The Reality Check: Remember the "Floor" for two different humans? That was 0.626.
The Conclusion: The robot's writing is actually less like the target human author than two random strangers are like each other. The robot is stuck in its own "robot voice" and cannot cross the gap to sound human.

3. The Trap: Why the Old Tools Lied

The paper explains why the old methods (like asking another AI to judge the writing) made the robot look good when it wasn't.

The "Circular" Trap: Imagine a teacher (the Judge AI) asks a student (the Robot) to write a story about "cats." The teacher then grades the story based on how well it mentions "cats."
- The robot was trained to listen to the teacher's instructions perfectly. So, it wrote a story full of "cats" and got a perfect score.
- But the teacher didn't ask if the story sounded like a specific human author. The robot just followed instructions.
- The Analogy: It's like a chameleon that changes color to match the instruction "be green," but the test was actually "be a specific person's skin tone." The robot passed the instruction test but failed the identity test.

4. The "Uncanny Valley" of Style

The researchers found that while the robot can change its style slightly to match different targets (it can sound a bit more like Author A than Author B), it never leaves its own "robot neighborhood."

The Neighborhood Analogy: Imagine all human writers live in a city called "Humanity." The robot lives in a separate city called "Robotia."
The robot can build a small bridge toward the Human city, but it never actually crosses the border. Even when it tries its hardest, it is still living in Robotia. The "Authorship Gap" is the wide river between the two cities that the robot cannot swim across.

5. The Lesson: Don't Trust the "Feeling"

The paper concludes that if you don't use a scientific, calibrated ruler (like the one they used, called LUAR), you might think you are making progress when you aren't.

The Takeaway: Just because a robot follows your instructions or uses the right keywords doesn't mean it has captured a human's soul or style. To truly know if we are succeeding, we need to stop guessing and start measuring with tools that have been proven to work for decades.

In short: We thought we were teaching robots to sound human, but we were just teaching them to follow orders. When we finally measured the "human-ness" correctly, we found the robots are still sounding like robots.

1. Problem Statement

The field of Large Language Model (LLM) personalization currently lacks a rigorous evaluation methodology for stylistic personalization (making an LLM write in a specific individual's voice). Existing benchmarks focus on:

Task Accuracy: (e.g., LaMP)
Preference Alignment: (e.g., PersonalLLM)
Conversational Persona: (e.g., PersonaLens using LLM-as-judge)

The Gap: None of these methods evaluate authorship fidelity—whether the generated text actually possesses the underlying "fingerprint" of the target human author. Current metrics are "ad hoc," lacking theoretical grounding in authorship science, which leads to uncalibrated scores and potentially misleading conclusions about model performance.

2. Methodology

The authors propose a theory-grounded evaluation framework that contrasts established authorship verification theory with ad hoc alternatives.

Data and Setup

Corpus: Blog Authorship Corpus (681K posts, 19k bloggers).
Subset: 50 authors with sufficient data (≥200 training, ≥50 test posts).
Generations: 1,000 total generations (50 authors × 5 prompts × 4 methods).
Prompting: Uses neutral content summaries (LLM-extracted) to avoid prompt contamination (e.g., leaking the author's first sentence).
Models: Primary generator: Qwen 3 (32B); Replication: GLM-4 (32B).

Four Inference-Time Personalization Methods Tested

Non-Personalized: Control group (content summary only).
Few-Shot: 5 author samples provided without explicit instruction.
Profile Extraction: Two-stage process (extract abstract style profile $\rightarrow$ generate from profile).
Contrastive: Author samples + contrastive examples from other authors + stylometric features.

Three Evaluation Metrics

The study evaluates these methods using three distinct metrics from different traditions:

LUAR (Primary - Theory-Grounded): Learning Universal Authorship Representations. A transformer model trained via contrastive learning on millions of Reddit posts to produce author-discriminative embeddings.
- Metric: Cosine similarity between aggregated embeddings of generated text and real author text.
- Advantage: Provides calibrated baselines (Human Ceiling and Cross-Author Floor).
LLM-as-Judge (Secondary - Ad Hoc): A decoupled binary protocol using GLM-4.
- Process: Extract 5 style traits per author $\rightarrow$ Score generation on traits $\rightarrow$ Judge same-author plausibility.
- Metric: Trait Match Rate (TMR).
Automated Stylometrics (Tertiary - Classical):
- Metric: Function word cosine similarity (FuncCos) over 60 common function words (Argamon et al., 2003).

3. Key Contributions

Calibrated Baselines for Authorship: The paper introduces the first evaluation framework for stylistic personalization grounded in authorship verification theory, establishing absolute score meanings (Ceiling: 0.756, Floor: 0.626).
Discovery of the "Authorship Gap": It demonstrates that current inference-time personalization methods fail to shift the LLM's stylistic fingerprint toward the target human, leaving generated text significantly more distant from the target author than random humans are from each other.
Construct Validity Failure: The study proves that ad hoc metrics (LLM judges) and classical metrics (stylometrics) measure different constructs than authorship verification, leading to near-zero correlations ( $|r| < 0.07$ ) and contradictory conclusions.
Identification of Circular Evaluation: It exposes how "Profile Extraction" methods artificially inflate LLM-judge scores by optimizing for the very traits the judge extracts, rather than genuine stylistic fidelity.

4. Key Results

The Authorship Gap

LUAR Scores: All four personalization methods scored between 0.484 and 0.508.
The Floor: The "Cross-Author Human Floor" (similarity between two different humans) is 0.626.
The Ceiling: The "Same-Author Human Ceiling" is 0.756.
Conclusion: All LLM-generated text scored below the human cross-author floor. This means personalized LLM output is stylistically less similar to the target author than a random human is to another random human. The LLM's own "fingerprint" dominates the output.

Metric Disagreement & Validity

Correlation: The three metrics showed near-zero pairwise correlations ( $|r| < 0.07$ ).
The "Winner" Illusion: The LLM-as-Judge (TMR) declared Profile Extraction the clear winner ( $d=0.58$ effect size). However, LUAR found no meaningful differentiation between methods.
Reason for Discrepancy: Profile Extraction optimizes for LLM-extractable traits. Since the Judge also uses an LLM to extract traits, the method "gaming" the judge. Real authors scored lower (0.427) on TMR than the Profile Extraction method (0.542), proving TMR measures instruction-following, not authorship fidelity.
Stylometrics: FuncCos showed collapsed baselines (Ceiling 0.742 vs. Floor 0.695), failing to distinguish same-author from cross-author text effectively.

Robustness

Cross-Model: Replication with GLM-4 confirmed the gap persists (scores 0.417–0.576, still below the 0.626 floor).
Embedding Space: LLMs occupy a distinct region in LUAR embedding space. While personalization modulates output within the LLM's style space (Gen $\leftrightarrow$ Gen AUC=0.918), it cannot bridge the gap to the human manifold (Gen $\leftrightarrow$ Real AUC $\approx$ 0.45–0.49).

Methodological Confound

Prompt Contamination: Using the first sentence of a target post as a prompt inflated the unpersonalized baseline similarity by 28 percentage points, masking the true performance gap. Neutral content summaries are required for valid evaluation.

5. Significance and Implications

Theory-Benchmark Cycle: The paper demonstrates a virtuous cycle where Authorship Theory (Stamatatos, Rivera-Soto) generates testable predictions (LLMs cannot fully erase their fingerprint), and Benchmarks (LUAR) confirm them with calibrated measurements.
Redefining Evaluation: It argues that task accuracy and preference alignment are insufficient for stylistic personalization. Future benchmarks must use theory-grounded metrics with calibrated baselines to avoid "hallucinated" progress.
Structural Limitation: The findings suggest a fundamental limitation of inference-time personalization. The LLM's inherent stylistic fingerprint is robust and cannot be overridden by prompting or few-shot learning alone; closing the gap may require training-time approaches (e.g., LoRA adapters).
Diagnostic Tool: Metric disagreement should be treated as a signal of construct validity failure rather than an opportunity to average scores.

In summary, the paper exposes a critical failure in current LLM personalization research: while models can follow instructions to mimic traits, they cannot currently replicate the deep, structural authorship fingerprint of a specific human, a fact invisible to uncalibrated, ad hoc evaluation metrics.