Interpretable Predictability-Based AI Text Detection: A Replication Study

This paper presents a replication and extension of the AuTexTification 2023 authorship attribution system, demonstrating that integrating newer multilingual models with document-level stylometric features and SHAP analysis improves detection performance across languages while highlighting the critical need for clear documentation to ensure reliable replication.

The Gist

Imagine you are a detective trying to solve a mystery: Who wrote this story? Was it a human with a unique voice, or was it a sophisticated robot (AI) mimicking a human?

This paper is like a "detective's field report" where the authors tried to rebuild and improve a specific tool used to solve this mystery. They looked at a system from a 2023 competition, tried to copy it exactly, found some things were missing or broken, and then built a better, more transparent version.

Here is the breakdown of their journey, using simple analogies:

1. The Mission: Copying the Blueprint (Replication)

The authors started by trying to rebuild a machine built by other researchers in 2023. Think of it like trying to bake a cake using a recipe and a photo of the finished product, but without the original chef's kitchen.

• The Problem: When they tried to bake the cake, it didn't taste exactly the same. Why?
  • Missing Ingredients: Some of the original "flour" (specific AI models for Spanish) had disappeared from the internet. They had to use a substitute.
  • Different Ovens: The original recipe said to bake for "until done," but the code they found baked for a fixed time.
  • Measurement Errors: The way they counted the "sugar" (linguistic features) was slightly different because the measuring cups (software tools) had changed over time.
• The Lesson: You can't truly copy a scientific experiment unless the original team shares every single detail, down to the exact version of the software they used.

2. The Upgrade: New Engines and Better Magnifying Glasses

Once they had a working machine, they decided to upgrade it. They wanted to see if newer technology could do a better job.

• The Engine (The AI Models): The original machine used an old engine (GPT-2) to guess how "predictable" a sentence was. The authors swapped this for newer, smarter engines (like Qwen and mGPT).
  • Analogy: Imagine the old engine was a bicycle trying to keep up with a car. The new engines are high-performance sports cars. They can spot the subtle differences between human and AI writing much faster and more accurately.
• The Magnifying Glass (Stylometric Features): The original tool looked at the text through a standard magnifying glass. The authors added 26 new lenses (features) to look at the text more closely.
  • They started counting things like: How many exclamation points are there? How long are the sentences? How repetitive are the words?
  • Analogy: If the AI is a forger, it might write perfect grammar, but it might use the same sentence structure over and over, or avoid complex emotional words. These new lenses catch those tiny "tells" that the old tool missed.

3. The Result: One Tool for All Languages

The original system needed a different tool for English and a different one for Spanish. It was like having two different keys for two different locks.

• The Breakthrough: The authors created a "Master Key" (a multilingual configuration). They found a single setup that worked just as well for English as it did for Spanish.
• Why it matters: This saves time and money. Instead of building a custom detector for every language in the world, we can use one smart, adaptable detector for many languages.

4. The "Why" (Interpretability)

Most AI detectors are "Black Boxes." You put text in, and it spits out a result, but you have no idea why. It's like a judge saying "Guilty" without explaining the evidence.

• The SHAP Analysis: The authors used a technique called SHAP (which stands for "Shapley Additive exPlanations"). Think of this as a highlighter pen that lights up exactly which parts of the text made the AI suspicious.
• The Discovery: They found that the new "lenses" (the 26 extra features) were actually very important. The AI wasn't just guessing; it was paying attention to things like sentence length and word variety. This makes the system trustworthy because we can see the evidence.

Summary: What Did They Learn?

1. Reproducibility is hard: If scientists don't share their exact code and settings, others can't verify their work. Small details matter.
2. Newer is better: Using modern AI models to analyze text makes detection much stronger.
3. Style matters: Even with powerful AI, looking at the "style" of writing (how sentences are built, punctuation, etc.) is still a superpower for catching fakes.
4. One size fits all: We can build one smart detector that works for multiple languages without needing to tweak it for each one.

In a nutshell: The authors took an old, slightly broken detective kit, fixed the missing parts, added high-tech lenses, and created a single, transparent tool that can catch AI writers in both English and Spanish, while showing us exactly how it caught them.

Technical Summary

1. Problem Statement

The rapid advancement of Large Language Models (LLMs) has made machine-generated text increasingly indistinguishable from human writing, posing significant challenges in education, journalism, and security. While early research focused on binary detection (Human vs. AI), the field has shifted toward model attribution (identifying the specific generator).

However, existing approaches face three main issues:

1. Reproducibility: Many studies lack sufficient detail or code to allow exact replication of results.
2. Black-Box Nature: State-of-the-art neural models often lack interpretability, making it difficult to understand why a decision was made.
3. Language Specificity: Many systems require separate configurations for different languages, hindering scalability.

This paper addresses these issues by replicating and extending the hybrid system from the AuTexTification 2023 shared task, focusing on reproducibility, the impact of base model selection, and the integration of interpretable stylometric features.

2. Methodology

The study follows a three-phase experimental design addressing three Research Questions (RQs):

Phase 1: Replication (RQ1)

• Goal: Reproduce the original system by Przybyla et al. (2023).
• Setup: The authors attempted to replicate the "Hybrid" and "Hybrid+" architectures, which combine:
  • Predictability Features: Token-level probabilities derived from generative models (originally GPT-2).
  • Contextual Representations: Fine-tuned Language Models (FLM) using RoBERTa.
  • Linguistic Features: Word frequency, grammar correctness, and document-level statistics.
• Challenges: Exact numerical replication was impossible due to:
  • Differences in data splits (topic-based vs. random).
  • Unavailability of original Spanish GPT-2 models, requiring substitution.
  • Ambiguities in feature extraction (e.g., context window sizes, spaCy library versions).
• Outcome: The authors successfully reproduced the pipeline and identified specific technical factors (data splits, model versions) that cause performance variance, highlighting the need for rigorous documentation.

Phase 2: Base Model Impact (RQ2)

• Goal: Determine if a unified multilingual configuration can replace language-specific models without sacrificing performance.
• Changes:
  • Generative Models: Replaced GPT-2 with newer multilingual models (Qwen, mGPT, LLaMA, BLOOM) to compute probabilistic features.
  • Encoders: Replaced language-specific RoBERTa with multilingual encoders (XLM-RoBERTa, mDeBERTa-v3).
• Selection: The authors evaluated three model groups (XGLM, mGPT, Large) and selected mDeBERTa-v3 as the shared encoder and the Large group (including Qwen2.5-3B, LLaMA-3.2-3B) for probabilistic features.

Phase 3: Stylometric Extension (RQ3)

• Goal: Enhance performance and interpretability by adding document-level stylometric features.
• Extension: Added 26 new features across seven categories:
  • Lexical diversity (e.g., Type-Token Ratio).
  • Sentence structure (e.g., length variance).
  • Repetition patterns (bigram/trigram repetition).
  • Functional markers (hedging words, transition words).
  • Readability metrics and punctuation usage.
• Analysis: Used SHAP (SHapley Additive exPlanations) to analyze feature importance and interpret model decisions.

3. Key Contributions

1. Systematic Replication: A critical analysis of the AuTexTification 2023 system, identifying that minor implementation details (data splits, library versions) significantly impact reproducibility.
2. Unified Multilingual Configuration: Demonstrated that a single configuration using recent multilingual models (mDeBERTa-v3 + Large generative group) achieves performance comparable to or better than language-specific baselines for both English and Spanish.
3. Enhanced Interpretability: Extended the feature set with 26 stylometric indicators and proved via SHAP analysis that these features are not only effective but also provide transparent insights into model decision-making.
4. Open Science: The implementation code is publicly available to ensure future reproducibility.

4. Key Results

• Replication: Exact numerical reproduction was not achieved (differences of 0.008–0.070 F1 points), primarily due to data split differences and model unavailability. However, the core trends were preserved.
• Multilingual Performance: The unified multilingual configuration (mDeBERTa-v3 + Large group) achieved an average Macro F1 of 0.717 (Hybrid) and 0.675 (Hybrid+), outperforming the language-specific baseline in the Hybrid configuration and matching it in Hybrid+.
• Stylometric Impact:
  • Adding the 26 new features significantly improved the LingRF (linguistic-only) model, with gains up to +21.6% for Spanish Subtask 1.
  • In the combined LingRF + PredOut model, the extended features provided the best overall results (e.g., 0.912 F1 for English Subtask 1), outperforming the state-of-the-art detector mdok in binary detection tasks.
• Interpretability: SHAP analysis confirmed that the new stylometric features (lexical diversity, sentence structure, functional markers) are among the most important predictors, complementing the probabilistic signals rather than replacing them.

5. Significance and Conclusion

This study underscores that feature design and implementation transparency are as critical as model architecture in AI text detection.

• Practicality: It proves that a single, unified multilingual system can effectively handle multiple languages and subtasks, reducing the engineering overhead of maintaining language-specific pipelines.
• Interpretability: By integrating stylometric features, the system moves beyond "black box" predictions, allowing researchers to identify specific stylistic markers (e.g., repetition, sentence complexity) that distinguish AI from human text.
• Future Work: The authors suggest that while neural models are powerful, they should be complemented by interpretable linguistic features. Future research should focus on identifying universal stylometric features that generalize across different generators and domains.

The paper concludes that reliable progress in AI detection requires not just better models, but rigorous documentation, standardized data splits, and transparent feature engineering.