💻 computer science

The Base-Rate Trap in Generative AI Text Detection: Why Detectors Cannot Serve as Standalone Evidence of Academic Misconduct, and Who Bears the Cost

This paper argues that generative AI text detectors are structurally unsuitable as standalone evidence of academic misconduct due to the base-rate fallacy, which causes unacceptably high false-positive rates and severe disparate impacts on non-native English writers, necessitating a shift from detection-based policing to assessment redesign.

Original authors: TANZIM ISLAM KHAN

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: TANZIM ISLAM KHAN

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a detective in a city where almost everyone is innocent, but a few people are secretly wearing invisible cloaks that make them look exactly like the rest of the crowd. You have a special "Cloak Detector" gadget. The gadget's sales brochure says it is 90% accurate, which sounds amazing. But here is the catch: in a city of 10,000 people, maybe only 1,000 are wearing cloaks (10%), and the other 9,000 are definitely not. If your gadget makes a mistake just 10% of the time on innocent people, it will accidentally flag 900 innocent people as wearing cloaks. Meanwhile, it might only catch 900 of the actual cloak-wearers. Suddenly, your "90% accurate" gadget has pointed the finger at 1,800 people, but only half of them are actually guilty. This is the "Base-Rate Trap." It's a famous puzzle in science that shows how even good tools can fail miserably when the thing you are looking for is rare. This problem isn't just about magic cloaks; it's about how we use math to make big decisions in medicine, law, and now, in schools.

This is exactly the story a new paper by Tanzim Islam Khan from Monash University tells us about Artificial Intelligence (AI) detectors in schools. As AI tools like ChatGPT have become popular, schools are trying to use software to catch students who submit work generated by AI. The paper argues that these detectors are currently useless as the only proof that a student submitted AI-generated work, and using them that way is actually unfair and dangerous.

The author didn't test real students or real essays. Instead, he built a giant, super-smart computer simulation—a digital "what-if" machine. He fed it the best numbers we have from real-world tests of these AI detectors and the best estimates of how many students actually submit AI-generated work. Then, he ran the simulation thousands of times to see what would happen in a typical school year with 10,000 essay submissions.

Here is what the simulation found, and it's a bit of a shocker. Even if you use a detector that is considered "near-perfect" (with a score of 0.99 out of 1.0), and you assume that 10% of students submit AI-generated work, the detector will still be wrong more than it is right. In the simulation, when the detector flagged a student, there was only a 67.9% chance they were actually guilty. That means nearly one out of every three students accused was innocent. If the detector is a bit less perfect (which most real ones are), the situation gets much worse. At a realistic 10% rate of AI-generated work, a standard detector might only be right 22.6% of the time. In other words, for every one student it catches, it falsely accuses three innocent ones.

The paper explains that this isn't because the detectors are poorly built; it's a math problem. The paper proves that to be 95% sure that a flagged student is actually guilty (a standard usually required in serious disciplinary cases), the detector would need to be almost impossibly good—so good that it would have to ignore almost all the actual students submitting AI-generated work to avoid making mistakes. It's like trying to find a needle in a haystack by throwing away the whole haystack just to be safe; you won't find the needle, but you also won't hurt anyone.

The simulation also revealed a sad truth about fairness. The paper modeled how these detectors treat students who write in English as a second language. Because their writing style is sometimes more predictable (like a machine), the detectors get confused and think they are AI. The simulation showed that if a detector is set up to be fair to native English speakers, it will falsely accuse non-native speakers at a rate 16 times higher. In a typical school of 10,000 students, the simulation predicted that about 2,213 innocent students would be wrongly accused, with non-native speakers bearing the brunt of the blame.

The paper concludes that schools need to stop treating these detector scores as the "smoking gun" evidence of students submitting AI-generated work. The math simply doesn't allow it. Instead of trying to catch students submitting AI-generated work with a tool that is mathematically guaranteed to make thousands of mistakes, the author suggests schools should change how they give tests. They should design assignments that are harder to submit AI-generated work on in the first place, like oral exams or projects done over time. The paper insists that the safest way to protect students from being wrongly accused is to stop relying on the detectors to make the final call.

Technical Summary: The Base-Rate Trap in Generative AI Text Detection

Problem Statement
Higher education institutions increasingly rely on generative artificial intelligence (GenAI) text detectors to adjudicate academic misconduct, justifying adoption through reported accuracy metrics such as the Area Under the Receiver Operating Characteristic Curve (AUC). However, this paper argues that these metrics are fundamentally inappropriate for consequential decisions regarding individual students. The core issue is the base-rate fallacy: in academic settings, the prevalence of actual GenAI misuse (the prior probability) is low (plausibly 5–15% based on recent surveys), while the vast majority of submissions are genuine. Under these conditions, even detectors with high AUC values produce a high rate of false accusations when used as standalone evidence. The paper posits that the critical question for disciplinary proceedings is not "how accurate is the tool?" but "given a flag, what is the probability that misconduct actually occurred?" This posterior probability, known as the Positive Predictive Value (PPV), collapses as the base rate of misuse decreases, rendering current detectors unsafe for individual adjudication.

Methodology
The study employs a transparent, reproducible modeling exercise rather than empirical data collection from student writing. The methodology integrates three primary components:

Signal Detection Theory (SDT): A Gaussian equal-variance model maps detector scores to a continuous distribution where human text follows $N(0,1)$ and AI text follows $N(d',1)$ , with $d'$ representing discriminability. Reported AUC values are translated into $d'$ to derive True Positive Rates (TPR) and False Positive Rates (FPR) at various decision thresholds.
Bayesian Base-Rate Reasoning: The model applies Bayes' rule to calculate PPV and the False Discovery Rate (FDR) across a range of misuse prevalences ( $\pi$ ) and detector performances. It derives a distribution-free safety criterion: the Positive Likelihood Ratio ( $\Lambda^+ = TPR/FPR$ ) required to meet a specific evidentiary standard ( $\tau$ ).
Monte Carlo Simulation: To validate closed-form results and quantify uncertainty, the author simulates 2,000 replications of an institution-term with 10,000 submissions. This simulation incorporates an equity model that accounts for the documented tendency of perplexity-based detectors to misclassify non-native (L2) English writing as AI-generated. It also includes an expected-harm analysis weighing the asymmetric costs of false accusations ( $C_{FP}$ ) versus missed cases ( $C_{FN}$ ).

Key Contributions
The paper makes five specific contributions to the discourse on academic integrity and AI detection:

A Framework for PPV: It provides a transparent SDT and base-rate framework that maps any reported detector AUC and assumed misuse prevalence to the probability that a flag is correct, demonstrating that accuracy metrics are misleading for individual cases.
Quantification of the "Base-Rate Trap": It identifies the parameter space where detectors are effectively unusable as evidence, showing that current tools cannot meet defensible evidentiary standards (e.g., 95% PPV) without failing to detect the vast majority of actual misconduct.
A Distribution-Free Safety Criterion: The paper derives a mathematical inequality (Proposition 1) stating that to safely accuse at a given prevalence and standard, a detector must exceed a specific likelihood ratio. It proves that this requirement grows inversely with prevalence ( $\propto 1/\pi$ ), making it unattainable as misuse becomes rarer.
Fairness Analysis: It models the disparate impact on non-native writers, demonstrating that thresholds calibrated on native speakers impose a quantifiable, structural burden on L2 writers, with wrongful-flag rates reaching up to 16.7 times higher than for native writers.
Policy Recommendations: It offers a "deployment calculator" for institutions, advising them to compare a tool's likelihood ratio against the required threshold before adoption, and to shift resources from detection toward assessment redesign.

Results
The simulation and analytical results yield several critical findings:

Low Predictive Value: Under realistic conditions (AUC $\approx$ 0.80, prevalence 10%), the PPV of a flag is only 22.6%. This means more than three in four flagged students are innocent. Even with a near-perfect detector (AUC 0.99), the PPV at 10% prevalence is only 67.9%.
The Detection Ceiling: To achieve a 95% evidentiary standard (PPV $\ge$ 0.95) at 10% prevalence, a detector with AUC 0.95 can catch at most 26.3% of true cases. A realistic detector (AUC 0.80) cannot reach this standard at any meaningful detection rate.
Disparate Impact: When a threshold is set to a 5% FPR for native writers, non-native writers (modeled with a shift calibrated to empirical studies) experience an 83.3% wrongful-flag rate, a 16.7-fold disparate impact.
Expected Harm: In a typical deployment simulation (AUC 0.80, 10% prevalence), the model predicts roughly 2,213 wrongful accusations per 10,000 submissions.
Likelihood Ratio Gap: Meeting a 95% standard at 10% prevalence requires a likelihood ratio of 171. Even a near-perfect detector (AUC 0.99) delivers a likelihood ratio of only 19.0, falling short by nearly an order of magnitude.

Significance and Claims
The paper concludes that the inability of detectors to serve as standalone evidence is a structural property of the decision problem, not a defect of specific tools that can be fixed by better engineering. The "base-rate trap" implies that as long as misuse remains a minority behavior, automated flags will overwhelmingly be false positives.

The author claims that:

Detection scores should never be the sole or primary evidence of misconduct, as doing so inverts the presumption of innocence.
Institutions must shift resources from detection toward assessment redesign (e.g., authentic, personalized tasks, process evidence) and prevention, as "AI-proof" assessment is a myth.
Reporting metrics must change: Institutions should evaluate tools based on prevalence-conditioned predictive values and likelihood ratios rather than accuracy or AUC.
The cost of false accusations is disproportionately borne by vulnerable groups (non-native speakers, neurodivergent writers), making the current reliance on detectors not just unreliable but discriminatory.

The study asserts that the safest number of students wrongly accused by an unreliable detector is zero, and the surest way to achieve this is to stop relying on the detector for adjudication.

Technical Summary: The Base-Rate Trap in Generative AI Text Detection

More like this