💬 NLP

GerAV: Towards New Heights in German Authorship Verification using Fine-Tuned LLMs on a New Benchmark

This paper introduces GerAV, a comprehensive German authorship verification benchmark comprising over 400k labeled text pairs from Twitter and Reddit, and demonstrates that fine-tuned large language models significantly outperform existing baselines and GPT-5 while revealing a trade-off between specialization and generalization across different data domains.

Original authors: Lotta Kiefer, Christoph Leiter, Sotaro Takeshita, Elena Schmidt, Steffen Eger

Published 2026-04-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Lotta Kiefer, Christoph Leiter, Sotaro Takeshita, Elena Schmidt, Steffen Eger

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective trying to solve a mystery: Did two different people write these two messages, or was it the same person?

This is the core of Authorship Verification (AV). In the real world, this helps police catch criminals who hide behind anonymous accounts, or helps fact-checkers spot if a fake news story was written by a known propagandist.

For a long time, most of the research on this "digital detective work" has been done in English. It's like having a massive library of clues in English, but almost no books in German. This paper, called GerAV, is like building that missing library and testing the best new detective tools on it.

Here is the story of the paper, broken down into simple parts:

1. The New Case File: GerAV

The researchers created a giant new dataset called GerAV. Think of this as a massive "mugshot book" of German writing styles.

The Size: It contains over 400,000 pairs of text messages.
The Sources: They grabbed posts from Twitter (short, punchy, like a tweet) and Reddit (longer, more conversational, like a forum).
The Twist: They didn't just dump everything in a pile. They organized it into different "training rooms":
- Same Topic: Two posts about cooking (easy to spot if the writer is the same).
- Different Topics: One post about cooking, one about politics (harder, because the content changes, so the detective must look only at how the person writes).
- The Whole Profile: Instead of one post, they gave the detective the author's entire history (like reading someone's whole diary vs. just one sentence).

2. The Detective Tools: Old vs. New

The researchers tested two types of detectives:

The Old School Detectives (Baselines): These are traditional methods. Some look at word patterns (like counting how often someone uses the letter "e"), while others use older AI models. They are like detectives who rely on a magnifying glass and a fingerprint kit.
The Super Detectives (LLMs): These are Large Language Models (like the AI you might chat with). The researchers took powerful, open-source AI models (like Gemma and Llama) and gave them a crash course on the GerAV dataset. This is called fine-tuning. It's like taking a genius who knows everything about language and teaching them specifically how to spot style rather than just facts.

3. The Big Showdown: Who Wins?

The results were clear: The Super Detectives won.

The Champion: A fine-tuned version of Gemma-3-12b (a specific AI model) became the best detective. It scored a 0.83 on a scale of 0 to 1.
The Gap: The best "Old School" detective only scored 0.74. That might sound small, but in the world of AI, that's a huge gap.
Beating the Giants: The researchers even tested the closed-source giant GPT-5 (a very famous, expensive AI). Even GPT-5, when asked to guess without any special training (zero-shot), was beaten by the researchers' smaller, free, fine-tuned model.

4. The Tricky Clues: What Makes it Hard?

The paper found some interesting "gotchas" in the investigation:

The Length Problem: If the text is too short (like a 10-word tweet), it's like trying to identify a singer by hearing only one note. It's very hard. The AI gets much better as the text gets longer (up to about 900 words).
The "Specialist" Trap: If you train a detective only on Reddit cooking posts, they become amazing at spotting cooking writers but terrible at spotting political writers.
- The Fix: The best strategy was to train the AI on a mix of everything (Twitter, Reddit, cooking, politics). This created a "generalist" detective who could handle almost any situation, even if they weren't perfect at every single specific task.
Language Matters: When they tried to use an AI trained on English data to solve German cases, it failed miserably. It's like trying to solve a German mystery using a dictionary that only has English words. You need a German-trained detective.

5. Why This Matters

This paper is a big deal for three reasons:

It fills the gap: We finally have a massive, high-quality benchmark for German authorship verification.
It proves AI is ready: Fine-tuned open-source AI models are now better than traditional methods and can even beat massive, closed-source giants like GPT-5 on specific tasks.
It's practical: Because these models are open-source, police forces, researchers, and security teams can actually use them locally without paying huge fees or worrying about data privacy.

In a nutshell: The researchers built a massive German "style library," trained a new generation of AI detectives on it, and proved that with the right training, these AIs can spot a fake author faster and more accurately than ever before. They showed that while content (what you say) changes, your unique writing fingerprint (how you say it) is hard to fake.

1. Problem Statement

Authorship Verification (AV) is the task of determining whether two texts were written by the same author. While extensively studied for English, there is a significant scarcity of large-scale benchmarks and systematic evaluations for other languages, particularly German. Existing methods often rely on feature-based approaches or neural models that lack the capacity of Large Language Models (LLMs). Furthermore, current evaluations often fail to address critical real-world forensic challenges, such as:

Cross-domain shifts: Determining authorship when texts come from different topics or platforms.
Text length variability: Handling everything from short tweets to long profile histories.
Language specificity: The reliance on English-centric models which may not capture German-specific stylistic nuances.

2. Methodology

A. The GerAV Benchmark

The authors introduce GerAV, a comprehensive benchmark comprising over 400,000 labeled text pairs derived from Twitter and Reddit. The dataset is structured into five distinct subsets to enable controlled analysis:

Reddit In-Domain (rid): Pairs from the same subreddit (topic).
Reddit Cross-Domain (rcd): Pairs from different subreddits (different topics), creating a harder evaluation scenario.
Reddit Profile-Based (rpb): Concatenated posts from a single user's profile to simulate having more data per author.
Twitter (tw): Short-form posts from German Twitter.
Mixed (mix): A combination of all above sources to test generalization.

Preprocessing: Data was filtered for German language (using langid), bot accounts were removed, and posts were constrained to 25–1,000 words (Twitter capped at 300 chars) to ensure quality and reduce noise.

B. Experimental Setup

The study evaluates three categories of models:

Established Baselines: Feature-based (n-grams, PPM compression), Neural (hLSTM, Siamese BERT), and recent style embedding models (mStyleDistance, msr).
Zero-Shot LLMs: Evaluation of Gemma-3, Llama-3, Qwen-2.5, and GPT-5 using standard prompts and Linguistically Informed Prompts (LIP).
Fine-Tuned LLMs: The authors fine-tuned four open-source LLMs (Gemma-3-12b, Llama-3.1-8b, Llama-3.2-3b, Qwen-2.5-7b) using LoRA (Low-Rank Adaptation) on the GerAV training splits.

Evaluation Metrics: Accuracy, F1-Score, and ROC AUC. Statistical significance was determined via paired non-parametric bootstrap tests.

3. Key Contributions

GerAV Benchmark: The creation of the largest German AV benchmark to date, featuring diverse data sources, domains, and text lengths, specifically designed to mimic forensic scenarios.
Comprehensive Baseline Comparison: The most extensive AV evaluation to date, comparing traditional baselines, zero-shot LLMs, and fine-tuned LLMs.
State-of-the-Art Performance: Demonstration that fine-tuned open-source LLMs significantly outperform existing baselines and even proprietary models like GPT-5 in specific settings.
Insights into Generalization vs. Specialization: A systematic analysis of how training on specific data types (e.g., cross-domain vs. in-domain) affects model performance across different test regimes.

4. Key Results

A. Performance of Fine-Tuned LLMs

Top Model: Gemma-3-12b, fine-tuned on the Mixed dataset, achieved the highest average F1-Score of 0.83 across all test splits.
Comparison to Baselines: This outperformed the best traditional baseline (msr, F1=0.74) by 0.09 and the best zero-shot baseline (Llama-3.2, F1=0.66) by 0.17.
Comparison to GPT-5: On a small stratified subset, the fine-tuned Gemma-3-12b (mix) outperformed GPT-5 by 0.08 F1 points (0.83 vs. 0.75), despite GPT-5 being a much larger, closed-source model.

B. Data Source and Domain Effects

Mixed Training: Models trained on the Mixed dataset showed the best generalization, suggesting that exposure to stylistic variability across domains is crucial for robust AV.
Cross-Domain Challenge: Performance dropped by 0.06 F1 points on cross-domain test sets compared to in-domain sets, indicating models still rely partially on topical cues. However, training on mixed data mitigated this better than training solely on cross-domain data.
Profile-Based: Evaluation using concatenated profiles (rpb) yielded the highest scores (up to 0.93 F1 for Gemma-3), confirming that more text per author significantly aids verification.

C. Text Length Sensitivity

There is a strong positive correlation (Spearman $\rho \approx 0.94$ ) between text length and F1-score.
Short texts (<25 words) are difficult to verify (F1 $\approx$ 0.78), while longer texts (>900 words) approach near-perfect accuracy (F1 $\approx$ 1.0).
Fine-tuned models only benefited from longer texts if the training data also contained long examples.

D. Low-Resource and Cross-Lingual Scenarios

Data Reduction: Reducing training data to 25% (20,000 samples) resulted in only a 0.04 drop in F1-score, indicating the approach is viable for low-resource settings.
Cross-Lingual Failure: A model trained on English Twitter data dropped significantly (F1 0.70 $\to$ 0.68) when tested on German data, and vice versa. This highlights the necessity of language-specific fine-tuning.
German Pretraining: A German-only pre-trained model (LLäMmlein-7b) did not outperform English-pretrained models after fine-tuning, suggesting that the fine-tuning process effectively adapts models to language-specific stylistic features regardless of pretraining language.

5. Significance and Implications

Forensic Applicability: The GerAV benchmark provides a realistic testbed for forensic applications (e.g., identifying authors in darknet forums or Telegram groups) where data is often short, cross-domain, and non-English.
Open-Source Superiority: The results challenge the notion that closed-source, massive models are always superior for specialized tasks. Fine-tuned, smaller open-source models (Gemma-3-12b) can outperform GPT-5 while offering full data privacy and reproducibility.
Methodological Shift: The study advocates for moving away from purely feature-based or zero-shot approaches toward fine-tuned LLMs for authorship analysis, particularly when dealing with language-specific nuances.
Future Directions: The authors identify the need for better interpretability (explaining why a decision was made) and further exploration of cross-platform authorship identification.

In conclusion, GerAV establishes a new standard for German Authorship Verification, proving that fine-tuned LLMs trained on diverse, high-quality data can achieve state-of-the-art performance, outperforming both traditional baselines and leading proprietary models in specific forensic contexts.