Why AI-Generated Text Detection Fails: Evidence from Explainable AI Beyond Benchmark Accuracy

1. Problem Statement

The rapid adoption of Large Language Models (LLMs) has created a critical need for detecting AI-generated text, particularly in high-stakes contexts like academic integrity. While current detectors report high accuracy on benchmark datasets (e.g., PAN-CLEF, COLING), their reliability in real-world scenarios is questionable.

The core problem identified is validity vs. accuracy:

• The Hypothesis: Many detectors do not learn universal, stable signatures of "machine authorship." Instead, they exploit dataset-specific artifacts (e.g., specific genres, formatting styles, prompt structures, or collection pipelines).
• The Consequence: A detector may achieve near-perfect scores on a benchmark but fail catastrophically when applied to unseen domains, different writing styles, or newer LLMs (distribution shift).
• The Gap: Existing research focuses heavily on benchmark performance and black-box models, lacking deep interpretability to understand why models succeed or fail.

2. Methodology

The authors propose an interpretable detection framework that avoids reliance on opaque large language models, instead using classical machine learning grounded in linguistic feature engineering.

A. Feature Engineering

The system extracts 38 interpretable linguistic features at the document level, categorized into five dimensions:

1. Surface & Length: Character/word counts, sentence/paragraph counts, punctuation.
2. Lexical Diversity: Unique word counts, Type-Token Ratio (TTR), Hapax ratio, Word Entropy.
3. Syntactic Structure: Sentence complexity, clause ratios, POS (Part-of-Speech) diversity and n-gram variety.
4. Readability & Predictability: Flesch Reading Ease, Gzip compression ratio, lexical surprisal.
5. Discourse & Style: Stopword counts, discourse markers, sentiment, hedging, and figurative language scores.

B. Datasets

The study utilizes three distinct datasets to test generalization:

• PAN CLEF 2025: Essays, news, fiction (focus on obfuscated authorship).
• COLING 2025: Peer reviews, scientific papers, Wikipedia (diverse, informational domains).
• Ghostbuster: An independent dataset (student essays, creative writing) used exclusively for cross-corpus testing to ensure no data leakage.

C. Experimental Pipeline

1. Feature Selection: Recursive Feature Elimination with Cross-Validation (RFECV) is used to select optimal feature subsets for each dataset independently.
2. Model Training: Four classifiers are trained: Logistic Regression, SVM, Random Forest, and XGBoost.
3. Ensemble: An equal-weighted ensemble averages probabilities from the four base models to improve robustness.
4. Evaluation Protocols:
   • In-domain: Train and test on the same dataset.
   • Cross-domain: Train on one dataset (e.g., PAN), test on another (e.g., COLING).
   • Cross-generator: Test on AI text generated by models not seen during training (e.g., GPT-5, DeepSeek).
5. Explainability: SHAP (SHapley Additive exPlanations) is applied to analyze global feature importance and instance-level predictions (including False Positives and False Negatives).

3. Key Contributions

1. Empirical Validity Analysis: The paper demonstrates that high benchmark accuracy is a poor proxy for real-world detection capability. Models often distinguish datasets rather than authorship.
2. Interpretable Framework: A novel pipeline combining linguistic features, classical ML, and SHAP that provides both predictions and instance-level explanations without using black-box LLMs.
3. Diagnostic of Failure Modes: Using SHAP, the authors identify that the features driving predictions are dataset-dependent. For example, a feature critical for PAN CLEF (e.g., POS diversity) may be irrelevant for COLING (where Gzip ratio dominates).
4. Open-Source Tool: Release of a Python package that returns predictions alongside SHAP-based explanations for individual texts, aiding reproducibility and responsible use.

4. Key Results

A. In-Domain Performance

• The feature-based models achieved competitive results, with the SVM model reaching an F1 score of 0.9734 on the PAN CLEF benchmark, ranking second overall against systems using fine-tuned LLMs.
• Trade-off: Tree-based models (XGBoost, Random Forest) achieved the highest in-domain accuracy but suffered the most significant performance drops in cross-domain settings. Linear models (SVM, Logistic Regression) showed lower peak accuracy but better stability.

B. Generalization Failure (Cross-Domain & Cross-Generator)

• Domain Shift: When models trained on PAN CLEF were tested on COLING, F1 scores dropped drastically (e.g., XGBoost fell from 96.94 to 67.23).
• Generator Shift: Detectors failed to detect text from unseen generators (e.g., GPT-5.2), with False Negative Rates (FNR) reaching nearly 60% for some models.
• Conclusion: Detectors are sensitive to the specific stylistic distribution of the training generator, not a universal "AI signature."

C. Explainability Findings (SHAP Analysis)

• Feature Divergence: The most influential features differed completely between datasets.
  • PAN CLEF: Dominated by POS diversity.
  • COLING: Dominated by Paragraph Count and Gzip Compression Ratio.
• Root Causes of Errors:
  1. Formatting Artifacts: The model learned that "single-paragraph" texts are AI. Consequently, short human essays (1 paragraph) were falsely flagged (False Positives), while multi-paragraph AI texts were missed (False Negatives).
  2. Compression Ratio Instability: Gzip ratio was a strong predictor but was confounded by text length and domain. AI texts with high lexical diversity (mimicking human variety) had compression ratios similar to human text, leading to misclassification.
  3. Text Length Bias: Short texts (<200 words) were highly prone to error. The model struggled to extract sufficient linguistic signals from short inputs, leading to a bias toward predicting the AI class for short human texts.

5. Significance and Implications

• Redefining Detector Validity: The paper argues that a detector is only valid if its decision logic remains stable across domains. Current high-performing detectors fail this test.
• Beyond Benchmarks: Relying solely on leaderboard accuracy is dangerous. Institutions must demand cross-domain evaluation and instance-level explanations before deploying detectors.
• Practical Recommendations:
  • Detectors should not be used as sole evidence for academic misconduct.
  • Systems should enforce minimum text length thresholds to avoid errors on short inputs.
  • Future research must shift from maximizing benchmark scores to developing robustness-aware evaluation protocols and content-level semantic features rather than surface-level structural cues.

In summary, the paper provides compelling evidence that current AI text detectors are largely "dataset classifiers" rather than "authorship detectors." By using Explainable AI, the authors expose the fragile, artifact-dependent nature of these systems, advocating for a paradigm shift toward transparent, robust, and context-aware detection tools.