Algorithmic Penalty for Non-Native Voices: A Three-Arm Diagnostic Accuracy Audit of Five Commercial AI Text Detectors Against Pre-LLM Scientific Literature — Protocol for the AUDIT-AI Study

Technical Summary: The AUDIT-AI Study Protocol

Problem Statement
Commercial AI text detectors are increasingly deployed by academic journals and grant reviewers to screen scientific writing for AI authorship. However, empirical evidence suggests these tools suffer from significant algorithmic bias against non-native English speakers. A foundational 2023 study (Liang et al.) demonstrated that detectors misclassified over 60% of essays by non-native English speakers as AI-generated, compared to near-perfect accuracy on native US-authored essays. The prevailing hypothesis is that perplexity-based detection methods interpret the constrained linguistic expression, reduced lexical diversity, and formulaic structures common in non-native writing as "machine-like" (low perplexity/low burstiness).

Despite this evidence, a critical gap remains: no peer-reviewed study has tested whether the geographic origin of a corresponding author's institution—a practical proxy used by editors—predicts detector misclassification in long-form, peer-reviewed medical literature. Existing validation studies rely on student essays, short abstracts, or mixed-genre corpora, which do not reflect the editorial screening context of high-stakes scientific publishing.

Methodology
The AUDIT-AI study is a pre-registered, three-arm diagnostic accuracy audit designed to quantify this bias in cardiometabolic research. The study adheres to the STARD-AI 2025 guidelines and is classified as Non-Human Subjects Research (NHSR).

• Corpus Construction (n = 200 texts):

  • Arm A (Native): 50 top-cited cardiometabolic papers (Diabetes, Obesity, NAFLD, Cardiovascular Diseases) from 1995–2020, with first and corresponding authors affiliated with institutions in the US, UK, Canada, Australia, New Zealand, or Ireland.
  • Arm B (Non-Native): 50 papers from the same domains and epochs, with authors affiliated with institutions in the Middle East/North Africa, East Asia, South Asia, Latin America, or non-English-speaking Europe.
  • Arm C (AI Control): 100 AI-generated rewrites (Introduction and Discussion sections) of the Arm A and Arm B papers.
    • Pipeline: A "decoupled, air-gapped" three-stage pipeline ensures methodological rigor. Stage 1 uses LlamaParse (vision-language model) to extract structured data from PDFs. Stage 2 uses Claude Sonnet 4.6 to generate narrative sections based strictly on the extracted data, enforcing stylometric overrides (e.g., varied sentence length) to maximize perplexity and avoid "AI-like" stock transitions. Stage 3 involves quality assurance gates for word count and content fidelity.

• Detection Ensemble:

  • Five commercial detectors are evaluated: iThenticate (Turnitin), Originality.AI, GPTZero, Copyleaks, and Winston AI.
  • Each of the 400 narrative sections (200 papers × 2 sections) is scored twice by each detector within a 14-day window, yielding 4,000 total observations.

• Statistical Analysis:

  • Primary Outcome: Continuous AI-probability score (0–100%).
  • Primary Model: Linear mixed-effects regression analyzing the difference between Arm B and Arm A, controlling for section type (Introduction vs. Discussion), publication epoch, journal impact factor, and word count.
  • Power: The design (n=50 per arm) provides ~80% power to detect a medium effect size (Cohen's d = 0.5) for the primary directional hypothesis.

Key Contributions and Novelty
The study claims to advance the field in five specific ways:

1. Long-Form Pre-LLM Text: Unlike prior work on short essays or abstracts, this study tests detectors on the 600–1,500-word narrative prose (Introduction/Discussion) actually used in editorial screening.
2. Geographic-Institutional Proxy: It utilizes the editor's practical signal (institutional affiliation) rather than self-reported L1 status or TOEFL scores.
3. Specialty Lock: By restricting the corpus to cardiometabolic literature, it controls for vocabulary register and topic distribution, isolating the L1 disparity from genre effects.
4. Methodological Air-Gap: The AI arm uses a different vendor (LlamaParse) for extraction than the generator (Claude), preventing critiques of training-data contamination.
5. Pre-Registered Rigor: All hypotheses, sample sizes, and statistical models are locked prior to data collection to prevent threshold cherry-picking.

Anticipated Results and Hypotheses
The study does not present final results but outlines five pre-registered hypotheses:

• H1 (Primary): Mean AI-probability scores will be significantly higher for non-native institution papers (Arm B) than native papers (Arm A), with an expected absolute difference of 12–22 percentage points.
• H2: No single detector will achieve simultaneous sensitivity and specificity of ≥0.85 across both human arms.
• H3: Inter-detector agreement (Fleiss' κ) at the 50% threshold will be poor (<0.40).
• H4: The L1 disparity will be larger in Discussion sections than in Introductions, reflecting the greater stylistic expressiveness of the former.
• H5: Geographic origin will remain a significant predictor of AI scores even after adjusting for journal impact factor, epoch, and readability metrics.

Significance and Claims
The AUDIT-AI study positions itself as a critical accountability mechanism for the tools governing scientific integrity.

• If H1 is supported: The study will provide direct evidence that reliance on commercial detectors imposes a measurable "equity cost" on biomedical authors from non-native institutions, potentially leading to unjust flagging in journal submissions and grant reviews.
• If H1 is not supported: It will offer the first negative finding in a structured, medical-domain test of the L1 disparity, challenging the generalizability of the Liang et al. (2023) TOEFL essay results to professional scientific publishing.

Ultimately, the study aims to inform policy decisions regarding the deployment of detectors in journal workflows, grant peer review (specifically referencing NIH guidelines), and academic integrity adjudications. The authors commit to releasing all code, schemas, and de-identified outputs under a CC-BY 4.0 license to ensure reproducibility.