🤖 AI

Measuring Embedding Sensitivity to Authorial Style in French: Comparing Literary Texts with Language Model Rewritings

This paper investigates how well French language model embeddings capture and retain authorial stylistic features after rewriting, demonstrating that these signals persist and exhibit model-specific patterns, thereby offering new avenues for detecting authorship imitation.

Original authors: Benjamin Icard, Lila Sainero, Alice Breton, Evangelia Zve, Jean-Gabriel Ganascia

Published 2026-05-12

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Benjamin Icard, Lila Sainero, Alice Breton, Evangelia Zve, Jean-Gabriel Ganascia

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Question: Can AI "Fake" a Writer's Soul?

Imagine you have a group of famous chefs (like Proust, Céline, and Yourcenar). Each has a unique way of cooking: one uses long, complex recipes with fancy ingredients; another cooks fast, loud, and messy; the third is precise and historical.

Now, imagine a super-smart robot chef (an AI or Large Language Model) that tries to copy these human chefs. The robot is told: "Cook this specific dish (a bus ride in Paris), but make it taste exactly like Chef Proust."

The big question this paper asks is: If we take a bite of the robot's food, can we still taste the human chef's unique "flavor," or does the robot's own mechanical taste take over?

In the world of computers, "taste" is measured using embeddings. Think of embeddings as a digital fingerprint or a GPS coordinate for a piece of text. Every sentence gets a specific set of numbers that tells the computer where it lives in a giant map of language.

The Experiment: The "Bus Ride" Test

The researchers set up a controlled experiment using French literature:

The Base Recipe (The Topic): They started with a book by Stéphane Tufféry that tells the exact same story (a bus ride in Paris) but written in many different styles. This ensures the story is the same, so any differences are purely about style.
The Human Chefs: They took texts from three real French authors:
- Proust: Known for long, flowing, complex sentences.
- Céline: Known for short, punchy, spoken-sounding sentences.
- Yourcenar: Known for balanced, historical, and precise writing.
The Robot Chefs: They asked three different AI models (GPT, Mistral, and Gemini) to rewrite the "Bus Ride" story, trying to copy the style of Proust, Céline, and Yourcenar.

How They Measured the "Flavor"

The researchers didn't just ask humans to read the texts. Instead, they used a mathematical map (called UMAP) to plot where all these texts landed.

The Human Map: When they plotted the real human authors, their texts formed distinct, separate islands. Proust's texts were in one cluster, Céline's in another, and Yourcenar's in a third.
The Robot Map: When they plotted the AI-generated texts, they wanted to see if the robots managed to land on the same islands as the humans, or if they drifted off into their own "Robot Zone."

They measured dispersion, which is like measuring how spread out a group of people is in a room.

If the AI successfully copied the style, the AI texts should be clustered tightly with the human texts (low dispersion).
If the AI failed, the AI texts would scatter away from the humans (high dispersion).

What They Found

1. The Robots Can Mimic, But Not Perfectly
The study found that the AI models did capture some of the human authors' styles. The "Robot Proust" texts did land closer to "Real Proust" than to "Real Céline." However, the connection wasn't perfect. The AI texts were like a slightly blurry photocopy of the original; the "fingerprint" was there, but it was fainter.

2. Different Robots Have Different "Hands"
Just like how a human chef might be better at baking than frying, different AI models were better at copying different authors:

Mistral was the best at copying Proust's complex style.
GPT was surprisingly good at capturing Céline's messy, spoken style.
Gemini did well with Yourcenar's historical precision.

3. The "Taste" Changes Depending on Who You Ask
The researchers noticed something interesting: The way they measured the "flavor" mattered.

If they looked at surface features (like how many commas or short words were used), one robot looked like the best copy.
If they looked at deeper structural features (like the complexity of the sentence structure), a different robot looked like the best copy.

It's like judging a painting: If you only look at the colors, one artist wins. If you look at the brushstrokes, a different artist wins. The AI models preserved different parts of the human style depending on which part of the "fingerprint" you were checking.

The Conclusion

The paper concludes that AI embeddings are sensitive to human style, but the signal gets weaker and distorted after rewriting.

Think of it like a game of "Telephone."

Round 1: A human whispers a story to another human. The message stays mostly the same, but the voice changes slightly.
Round 2: A human whispers to a robot, and the robot whispers back. The robot gets the gist of the story, but the unique "voice" of the original human gets muddled.

The study shows that while AI can mimic the look of a famous author, the deep mathematical "soul" of that author's writing is only partially preserved. This means that in the future, we might be able to detect if a text was written by a human or an AI by looking at how "fuzzy" or "distorted" these style fingerprints are.

Important Note: The paper does not claim this technology is ready to catch AI cheaters in real-time or to be used in court. It is a preliminary study showing that the "style signal" exists but is fragile. The authors suggest this is just the beginning of understanding how AI handles human creativity.

Technical Summary: Measuring Embedding Sensitivity to Authorial Style in French

Problem Statement
While Large Language Models (LLMs) can convincingly imitate human writing styles, it remains unclear how much stylistic information is encoded in the embeddings of language models and how well these signals are retained after LLM rewriting. Existing embedding-based analyses often prioritize semantic tasks (e.g., topic modeling) over stylistic features. This paper investigates whether embeddings reliably capture authorial stylistic features in French literary texts and to what extent these signals persist when texts are rewritten by LLMs to imitate specific authors.

Methodology
The study employs a controlled experimental design using a dataset of 1,248 French literary texts, divided into three components:

Reference Corpus (Human-Authored):
- TUFFERY_REF: 96 texts from Stéphane Tufféry's Le style mode d'emploi, all sharing a single topic (a bus journey in Paris) but varying in style (including pastiches of major authors).
- STYLE_REF: 288 texts from three distinct authors—Proust (Du côté de chez Swann), Céline (Voyage au bout de la nuit), and Yourcenar (Mémoires d'Hadrien)—representing diverse stylistic profiles (structural complexity, spoken register, and historical anchoring, respectively).
Generated Corpus (LLM Imitations):
- STYLE_GEN: 864 texts generated by prompting three LLMs (GPT-4o, Mistral-large-2411, and Gemini-1.5-flash) to rewrite the TUFFERY_REF texts in the styles of the three authors in STYLE_REF. This creates a fixed-topic, variable-style dataset.
Vectorization and Analysis:
- All texts were encoded using 13 different embedding models (ranging from 384 to 1536 dimensions), selected for architectural diversity and performance on the Massive Text Embedding Benchmark (MTEB).
- Clustering Validation: K-means clustering (k=3) and Uniform Manifold Approximation and Projection (UMAP) were used to assess the separability of the three corpus groups (TUFFERY, STYLE_REF, STYLE_GEN) in embedding space.
- Sensitivity Metrics: The core analysis measures embedding dispersion (Euclidean distance from class centroids in 2D UMAP space) against stylistic feature shifts. Five stylistic feature families were tracked: Structural (word/sentence length), Morphosyntax (POS tags), Lexicon (Shannon entropy, letter frequencies), and Referentiality (Named Entity Recognition density).
- Correlation Analysis: Pearson correlations were computed between the shift in embedding dispersion ( $\Delta d$ ) and the shift in stylistic feature frequencies ( $\Delta f_s$ ) relative to the TUFFERY_REF baseline. This was performed for both human-authored texts (STYLE_REF) and LLM-generated imitations (STYLE_GEN).

Key Results

Embedding Sensitivity to Human Style: Embeddings reliably capture authorial stylistic features. Significant, though moderate-to-weak, correlations were found between embedding dispersion shifts and stylistic feature shifts.
- Proust: Broad sensitivity across Named Entities (NER), letters, structural features, and POS tags.
- Céline: Concentrated sensitivity on letter frequencies, NER, and entropy.
- Yourcenar: Dominated by sensitivity to NER (referential density).
Persistence After Rewriting: Stylistic signals persist after LLM rewriting, but they are attenuated and reweighted.
- Proust Imitations: Retained a broad profile but with reduced sensitivity in structural and letter features.
- Céline Imitations: Showed the highest stylistic fidelity among generated classes, preserving the spoken-register markers (letters) while overweighting referentiality.
- Yourcenar Imitations: Retained NER-based anchoring but with partial dilution and a shift toward letter frequencies.
LLM-Specific Patterns: Different LLMs preserve different aspects of authorial style. For instance, Mistral best preserved Proust's broad profile, while GPT showed higher fidelity to Céline's and Yourcenar's defining patterns compared to other models, despite GPT having lower overall "transfer accuracy" in a surface-level classifier.
Metric Divergence: The study highlights a divergence between surface-level style transfer accuracy (measured by character n-grams) and embedding-based stylistic fidelity. A model might score lower on surface accuracy but better preserve the deep structural or referential patterns that define an author's embedding profile.

Significance and Claims
The paper claims that its analytical results offer promising directions for authorship imitation detection in the era of language models. By tracing how specific stylistic features are preserved or altered through generative rewriting, the study complements traditional style transfer evaluations. It suggests that deviations from a human baseline in embedding space, specifically regarding stylistic transformation patterns, may serve as indicators of LLM-based imitation.

The authors remain modest regarding their findings, noting that:

The observed correlations are significant but moderate in magnitude.
The study is limited to French literary texts and a specific set of authors; generalization to other languages or non-literary genres requires further validation.
The inventory of stylistic features is tailored to the specific authors studied and may not be exhaustive.
The reliance on 2D UMAP for visualization introduces distortions that require further investigation.

Ultimately, the work provides a framework for quantifying the sensitivity of embedding vectors to style and the extent to which LLMs preserve or alter that sensitivity during style transfer.