💻 computer science

Benchmarking Stylometric Metrics for AI Authorship Verification: A Glass-Box Forensic Framework Across Language Models

This paper proposes a transparent "Glass-Box" forensic framework utilizing forty interpretable stylometric metrics to demonstrate that, despite lexical convergence, distinct and widening divergences in information-theoretic predictability, structural properties, and natural variability persist between human and AI-generated text across diverse datasets, offering a robust foundation for authorship verification.

Original authors: Abhay Srivastava, Lovish Bhatia, V.S. Pandey, Ajay K. Sharma

Published 2026-09-23✓ Author reviewed ⓘ

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Abhay Srivastava, Lovish Bhatia, V.S. Pandey, Ajay K. Sharma

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the quiet corners of language, every writer leaves a unique fingerprint. This is the core idea of stylometry, a field that has long studied how people write not just by what they say, but by how they say it. Humans have unconscious habits: the length of their sentences, the rhythm of their punctuation, the variety of words they choose, and the way they structure their thoughts. For decades, linguists have used these subtle patterns to identify authors of anonymous letters or disputed historical texts. Today, a new challenge has emerged. Powerful artificial intelligence systems can now generate text that sounds remarkably human, blurring the line between a person's voice and a machine's output. This ability raises urgent questions for schools, courts, and newsrooms: when a piece of writing appears, how can we know if a human wrote it or if a computer generated it? The stakes are high, involving academic integrity, intellectual property, and the spread of misinformation.

A team of researchers at the National Institute of Technology Delhi has taken a fresh approach to this problem. Instead of relying on complex, opaque computer programs that simply guess whether text is real or fake, they built a transparent, step-by-step framework to measure the specific differences between human and machine writing. They called this a "glass-box" method, meaning every step of the analysis is visible and understandable, unlike the "black-box" systems that hide their reasoning. The researchers examined two massive collections of text: one containing controlled academic essays and another with diverse, real-world writing from the internet, including content from the newest, most advanced reasoning models. By measuring forty different features of the writing, they discovered that while artificial intelligence is getting better at mimicking human vocabulary, it is simultaneously becoming more rigid and predictable in its structure.

The researchers began by testing their forty measurement tools on human writing alone to ensure the tools were reliable. They split human-written texts into two groups and checked if the tools could find any differences between them. The result was that the tools found almost no difference, proving that human writing maintains a consistent, natural shape regardless of the topic or the specific author. This established a stable baseline. When they then applied these same tools to compare human writing against text generated by artificial intelligence, clear patterns emerged. In the controlled academic setting, the biggest differences appeared in the overall distribution of words and the predictability of the text. The machine-generated text was statistically more predictable than human writing, and the variety of words used was different.

However, the story became more complex when the researchers looked at the newer, more diverse dataset containing the latest reasoning models. These are systems designed to think through problems step-by-step before answering. Here, the researchers found a paradox. The artificial intelligence had become much better at mimicking the statistical unpredictability of human language, closing the gap in how "surprising" the words were. Yet, in other areas, the gap actually widened. The machine writing became structurally more uniform and repetitive. While humans naturally vary their sentence lengths and structures to create rhythm and emphasis, the newer models produced text with a robotic consistency. This was especially true for the newest reasoning models, which tended to write much longer texts with a very steady, unchanging sentence structure.

One of the most striking findings concerned punctuation. In the wild, uncontrolled environment of the internet, human writers use a wide range of punctuation marks, including semicolons, exclamation points, and question marks, to express complex thoughts and emotions. The artificial intelligence models, however, largely avoided these marks. They stuck to periods and commas, creating a safer, more neutral tone. The researchers traced this to the way these models are trained to be helpful and harmless, which inadvertently strips away the emotional and structural variety found in natural human speech. The machines were not just writing correctly; they were writing too perfectly, lacking the natural "noise" and imperfections that characterize human expression.

The study also highlighted a critical difference in how these models handle variance. Human writing fluctuates; a writer might use a very long, complex sentence followed by a short, punchy one. This variation is a sign of a living voice. The artificial intelligence, even when trying to be creative, tends to flatten these fluctuations. It produces text where the sentence lengths and word choices stay within a narrow, consistent range. The researchers found that this lack of natural variation was a stronger indicator of machine authorship than the specific words chosen. In fact, as the models became more advanced, their ability to mimic human vocabulary improved, but their inability to mimic human structural variety became even more obvious.

The researchers concluded that the old ways of detecting fake text, which often focused on simple word counts or basic vocabulary checks, are no longer enough. The machines have learned to use the right words. The new frontier for detection lies in looking at the structure and the rhythm of the writing. The most reliable signs of a machine are not what it says, but how it says it: the repetitive sentence patterns, the avoidance of complex punctuation, and the lack of natural variation in the flow of ideas. The study suggests that future tools to identify artificial text must focus on these deeper structural fingerprints. As artificial intelligence continues to evolve, it may eventually learn to write with a human-like vocabulary, but it will likely struggle to replicate the imperfect, varied, and rhythmic chaos of a human mind at work. The key to telling them apart lies in listening for that natural, human noise.

Technical Summary: Benchmarking Stylometric Metrics for AI Authorship Verification

Problem Statement

The rapid advancement of Large Language Models (LLMs) has blurred the distinction between human-authored text (HuAT) and AI-generated text (AIGT), creating significant challenges for forensic investigation in domains such as academic integrity, intellectual property rights, and disinformation. Existing literature relies heavily on "Black-Box" detection approaches, typically utilizing deep learning classifiers (e.g., RoBERTa-based architectures). The authors identify two critical forensic limitations in these methods:

Lack of Interpretability: These models cannot explain why a text is flagged, rendering them insufficient for legal or high-stakes administrative proceedings where the burden of proof requires explainability.
Susceptibility: Such classifiers are brittle and easily evaded through adversarial attacks, obfuscation, and paraphrasing.

The paper posits that while LLMs are proficient at mimicking human writing, subtle stylistic signatures of machine authorship may persist. The core problem is to quantify the divergence between HuAT and AIGT using an interpretable, "Glass-Box" framework that moves beyond binary detection to analyze the specific linguistic features that differentiate the two.

Methodology

The study introduces a Glass-Box Forensic Framework comprising 40 interpretable stylometric metrics, organized into three computational tiers and eleven linguistic categories.

1. The 40-Metric Taxonomy

The metrics are stratified to capture different levels of linguistic analysis:

Tier 1 (Foundational Metrics, M1–M15): Surface-level features including lexical metrics (e.g., word length, type-token ratio), syntactic structure, punctuation analysis, and distance metrics (Burrows' Delta, Cosine Similarity, Jensen-Shannon Divergence).
Tier 2 (Intermediate Metrics, M16–M30): Features analyzing text flow and pattern, including Part-of-Speech (POS) patterns, n-grams, information-theoretic measures (entropy, perplexity), readability scores, and discourse markers.
Tier 3 (Advanced Metrics, M31–M40): Deep forensic indicators focusing on semantic coherence, advanced vocabulary, and specific AI-style markers (e.g., grammar error density, token predictability variance).

2. Datasets

The framework was evaluated on two distinct benchmark datasets to assess performance across controlled and "in-the-wild" scenarios:

PAN24: A controlled, balanced dataset with predefined topics and 13 standard AI models. It serves as a "stylometric laboratory" with low length variation.
PAN25: A diverse, unbalanced dataset representing realistic usage patterns, including new "Reasoning Models" (e.g., DeepSeek-r1, o3-mini) and web-sourced human text. It features high length variation and model heterogeneity.

3. Statistical Analysis

Rather than relying on binary classification accuracy, the study employs an effect-size-based framework:

Cohen's d: Used to quantify the magnitude of divergence between human and AI distributions in units of pooled standard deviation.
Similarity Percentage (Sim%): Measures relative distributional similarity.
Variance Divergence ( $\Delta(\sigma/\mu)$ ): Analyzes whether AI models replicate the natural variability (noise) of human writing or exhibit "robotic uniformity."
Cross-Dataset Consistency: Spearman rank correlation ( $\rho$ ) of effect sizes between PAN24 and PAN25 to determine the stability of metrics across domains.

Key Results

1. Baseline Stability

Validation experiments splitting human text into two groups confirmed that the 40 metrics are stable across human authors, with negligible effect sizes ( $|d| < 0.1$ ). This establishes that observed divergences in HuAT vs. AIGT comparisons are genuine stylistic signatures rather than random noise.

2. Asymmetric Evolution of AI Mimicry

The comparison between PAN24 and PAN25 reveals an asymmetric evolution in AI capabilities:

Convergence in Predictability (Tier 2): In PAN24, information-theoretic metrics like Perplexity (M23) showed a large effect size ( $d = 0.968$ ), indicating AI text was highly predictable. In PAN25, this gap narrowed significantly ( $d = 0.356$ ), suggesting modern models have learned to mimic human statistical "surprise."
Divergence in Structure and Variance (Tier 1 & 3): Conversely, structural and variance-based metrics widened.
- Sentence Uniformity (M39): The effect size for sentence uniformity increased from medium in PAN24 ( $d = -0.642$ ) to very large in PAN25 ( $d = -1.619$ ). Reasoning models, in their pursuit of logical coherence (Chain-of-Thought), exhibit "hyper-structure" and repetitive syntactic patterns.
- Lexical Convergence vs. Distributional Divergence: Contrary to the expectation of seamless blending, lexical richness metrics partially converged, while distributional gaps widened. Specifically, the lexical richness gap (Yule's K, M03) narrowed from a small effect in PAN24 ( $d = -0.385$ ) to a medium effect in PAN25 ( $d = -0.777$ ), indicating AI is becoming more repetitive relative to unconstrained human text, yet the overall distributional distance (JS Divergence, M15) surged to a "Very Large" effect ( $d = -1.503$ ). This confirms that while AI may be converging on certain lexical means, it relies heavily on distinct, non-human vocabulary distributions rather than matching human linguistic diversity.
- Punctuation "Safety" Filter: The use of semicolons (M10) shifted from negligible divergence in PAN24 ( $d = 0.142$ ) to a large discriminator in PAN25 ( $d = 1.416$ ), attributed to RLHF safety tuning that suppresses complex or expressive punctuation.

3. The "Variance Gap"

A consistent finding across both datasets is that LLMs match human means but fail to replicate human variance. AI text exhibits "mode collapse," characterized by unnaturally low internal variance in sentence length and structure, whereas human writing maintains high variability (noise) as a sign of truth.

4. Cross-Dataset Consistency

A strong Spearman rank correlation ( $\rho = 0.71$ ) between the effect sizes of PAN24 and PAN25 indicates a stable hierarchy of discriminative metrics. This suggests that the identified stylometric signatures are inherent to the generative architecture of LLMs rather than being artifacts of specific datasets.

Significance and Claims

The paper claims to provide a principled foundation for forensic analysis that is distinct from opaque black-box detectors. Its primary contributions are:

Interpretability: By quantifying specific feature-level divergences, the framework offers explainable evidence suitable for legal and academic contexts.
Identification of the "Paradox of Improvement": The study demonstrates that as AI models improve in local statistical predictability (mimicking human randomness), they simultaneously become more rigid in global structural execution. The "Reasoning-Structure Tradeoff" suggests that the drive for logical coherence in Chain-of-Thought reasoning forces models into repetitive, template-like syntactic patterns.
Shift in Detection Strategy: The authors argue that future detection systems must move away from Tier 1 mean-based metrics (which are converging) and prioritize Tier 2 structural variance and Tier 3 generative artifacts. The "Hallucination of Style"—the failure of LLMs to reproduce the inherent imperfections and variability of human expression—is identified as the most robust and stable discriminator.

The paper concludes that the boundary between HuAT and AIGT is not disappearing but shifting; while vocabulary gaps are closing, the structural and variance-based "machine signatures" are becoming more pronounced, particularly in models designed for reasoning.