Benchmark Evaluation of the WordBinary AI-Text Detection Model on 9,000 AI-Generated Texts and 2,000 Pre-2015 Human Academic-Paper Samples

Technical Summary: Benchmark Evaluation of the WordBinary AI-Text Detection Model

Problem Statement
The proliferation of large language models (LLMs) has created an urgent need for systems capable of distinguishing AI-generated text from genuine human writing. This task is complicated by the variability of both human and machine writing across disciplines, educational backgrounds, and generation parameters. A critical challenge in this domain is balancing high sensitivity to machine-generated content with a sufficiently low false-positive rate to protect human authors, particularly in academic and educational settings where incorrect classification can lead to severe reputational and disciplinary consequences. Existing detection methods often struggle with generalization across unseen generators, adversarial modifications, and "humanised" outputs, while performance metrics frequently rely on accuracy alone rather than controlled false-positive rates.

Methodology
This study presents a retrospective benchmark evaluation of the WordBinary AI-text detection model using a labelled dataset of 11,000 text samples. The evaluation design focused on internal performance measurement rather than comparative analysis against other detectors.

• Dataset Composition:
  • AI-Generated Corpus (9,000 samples): Comprising outputs from seven platform groups: Claude, Cohere, DeepSeek, Gemini, Mistral, OpenAI, and OpenRouter. The corpus included both "normal" AI outputs (4,400 samples) and "humanised" AI outputs (4,600 samples), the latter undergoing unspecified rewriting or humanisation procedures.
  • Human Academic Corpus (2,000 samples): Sourced from pre-2015 academic papers across five disciplines: computer science, economics, statistics, physics, and mathematics (400 samples per discipline). The pre-2015 cutoff was used as an operational provenance criterion to minimize the likelihood of modern LLM authorship.
• Evaluation Metrics: The study calculated standard binary classification metrics including sensitivity (recall), specificity, precision, negative predictive value, F1 score, balanced accuracy, and the Matthews correlation coefficient (MCC).
• Statistical Analysis: Observed proportions were accompanied by exact 95% Clopper-Pearson binomial confidence intervals. The analysis assumed statistically independent observations, though the author notes that clustering by source paper or prompt was not accounted for due to a lack of sample-level identifiers.
• Constraints: The evaluation did not include a comparative benchmark against other commercial or open-source detectors, nor did it assess multilingual performance, mixed-authorship documents, or AI-polished human writing.

Key Results
The WordBinary model achieved perfect classification on the supplied dataset:

• Overall Performance: The model correctly classified all 9,000 AI-generated samples as AI and all 2,000 human samples as human. This resulted in 9,000 true positives and 2,000 true negatives, with zero false positives and zero false negatives.
• Metrics: Observed sensitivity, specificity, precision, negative predictive value, F1 score, balanced accuracy, and overall accuracy were all 100%. The Matthews correlation coefficient was 1.000.
• Confidence Intervals: Exact 95% confidence intervals were calculated as 99.959%–100% for sensitivity, 99.816%–100% for specificity, and 99.966%–100% for overall accuracy. The one-sided 95% upper confidence bound for the false-positive rate was approximately 0.184%.
• Platform-Level Performance: All seven AI platform groups achieved 100% observed recall. Mean AI scores ranged from 98.56% (DeepSeek) to 99.98% (OpenAI), with a weighted mean of 99.50%.
• Humanisation Impact: Humanised AI samples were detected at 100% recall, though their mean AI scores (99.03%) were slightly lower than normal AI samples (99.99%).
• Human Text Scores: No human academic samples were flagged as AI. The highest individual AI score assigned to a human sample was 1.07%, with domain-level means ranging from 0.01% (Mathematics) to 0.31% (Economics).

Key Contributions

1. Comprehensive Platform Coverage: The study demonstrates that the WordBinary model maintains complete recall across a diverse set of seven major AI platform groups, including both normal and humanised outputs.
2. Robustness to Humanisation: The model successfully detected all humanised AI samples, suggesting resilience against post-generation rewriting procedures, despite a measurable reduction in aggregate AI scores.
3. False Positive Protection: The evaluation provides evidence that the model can achieve high AI recall without misclassifying formal, technical human academic writing from five distinct disciplines as machine-generated.
4. Rigorous Statistical Reporting: The paper provides exact binomial confidence intervals for a dataset with zero errors, offering a statistically sound interpretation of the "100% accuracy" claim that acknowledges the limitations of finite sample testing.

Significance and Claims
The author claims that the results demonstrate "complete classification of the supplied test set" and provide "strong evidence of performance under its represented conditions." The study establishes that WordBinary can effectively distinguish between the specific AI-generated and pre-2015 human academic texts provided in the benchmark.

However, the paper explicitly and modestly frames these findings. It states that the results do not establish universal 100% accuracy. The author cautions that the perfect score may reflect a close match between the test distribution and the model's training data. They emphasize that the study is an internal benchmark and that broader generalization requires independent external validation, documentation of decision thresholds, control for training-test contamination, and evaluation on mixed-authorship, multilingual, and AI-polished documents. The paper concludes that while the model shows promising results for multi-platform and humanised AI detection, independent, preregistered, and reproducible external validation is the necessary next step before broader claims of efficacy can be made.