Beating the Style Detector: Three Hours of Agentic Research on the AI-Text Arms Race

Technical Summary: Beating the Style Detector

Problem Statement
The paper addresses the "AI-text detection arms race," where Large Language Models (LLMs) increasingly close the gap between their generated text and the specific writing style of a target human author. While previous studies have examined whether humans can post-edit LLM drafts to sound like themselves, this work extends the inquiry to whether frontier LLMs can autonomously mimic a specific author's style without human intervention. Furthermore, it investigates whether these advanced mimics can evade detection by classifiers trained to distinguish human writing from AI generation. The study also serves as a methodological demonstration of "agentic research," aiming to reproduce and extend an empirical NLP study (Baumler et al., ACL 2026) in hours rather than weeks using an automated research harness.

Methodology
The research is structured into three primary phases: reproduction, extension, and adversarial evaluation.

1. Reproduction and Agentic Harness:

   • The authors utilized a released dataset (81 JSON logs, 486 task observations) from Baumler et al. [2] but lacked the original analysis code.
   • An agentic-research harness (inspired by Karpathy's autoresearch and Zaiss et al.) was deployed to re-implement the full statistical pipeline. The human investigator acted only as a "reviewer-in-the-loop," making high-level decisions (e.g., checking for data leakage) while the agent generated code, ran experiments, and fixed bugs.
   • The pipeline used LUAR-MUD [10], a frozen authorship-style embedding model trained on Reddit, to map texts to 512-dimensional vectors. Similarity was measured via cosine similarity.
   • Statistical analysis employed paired permutation tests ($n_{perm}=10,000$), Hedges' $g$ for effect sizes, and repeated-measures correlation (rmcorr).

2. Leakage-Free LLM-Mimic Benchmark:

   • The study introduced two new conditions: GPT-5.5 and Claude Opus 4.7.
   • A strict held-out protocol was implemented to prevent data leakage. For each participant, one unassisted control text was used as a "style demonstration" (shown to the LLM), while the second unassisted control was reserved exclusively as the evaluation target.
   • The LLMs were tasked with generating a draft from scratch based on the planning prompt and the single style demo. These drafts were compared against the held-out target, alongside the original human post-edits and the initial o4-mini drafts.
   • This resulted in 324 paired tasks per generator (648 total mimic drafts).

3. Detection and Adversarial Rewriting:

   • Detection: A linear Support Vector Machine (SVM) was trained on LUAR-MUD embeddings to classify text as "Human" (unassisted controls) vs. "AI" (generated drafts). The evaluation used GroupKFold (leave-authors-out) to ensure the classifier generalized to unseen authors.
   • Diagnostics: Six robustness checks were performed, including length-only baselines, PCA dimensionality reduction, and cross-LLM transfer tests, to determine if detection relied on stylistic features or artifacts like text length.
   • Adversarial Loop: An Opus 4.7 agent was placed in a feedback loop against a frozen SVM detector. Given the detector's margin score on its current draft, the agent performed $T=20$ rewrite iterations to minimize the margin (i.e., make the text appear more human) while preserving content and length constraints.

Key Results

• Reproduction: The agentic harness successfully reproduced all seven preregistered hypotheses and the headline correlation between perceived and embedding-measured self-similarity ($r=+0.244$, matching the original to three decimal places) in approximately three hours.
• Style Mimicry Performance:
  • Frontier LLMs significantly outperformed human post-editing. GPT-5.5 and Opus 4.7 closed 71–75% of the style gap to the same-author ceiling, compared to only 24% for human post-editing.
  • In per-task win rates against the human-edit threshold, Opus 4.7 won 76.9% of tasks and GPT-5.5 won 79.0%, while the original o4-mini draft won only 18.5%.
  • There was no statistically significant difference between Opus 4.7 and GPT-5.5 ($p=0.14$).
• Detection and Diagnostics:
  • A leave-authors-out linear SVM achieved high AUCs for all approaches: o4-mini (0.999), Human post-edit (0.971), Opus 4.7 (0.952), and GPT-5.5 (0.931).
  • Crucial Diagnostic Finding: The detection signal for GPT-5.5 was largely a length confound (AUC 0.880 for length-only baseline), whereas the Opus 4.7 signal was genuinely stylistic (length-only AUC 0.517).
  • Cross-LLM transfer tests confirmed a generic "frontier-LLM signature" exists, but the specific stylistic overlap of Opus with human controls is distinct.
• Adversarial Success:
  • In the adversarial rewriting experiment, the Opus agent successfully reduced the detection margin for all five held-out test targets over 20 iterations.
  • Two of the five targets crossed the SVM decision boundary into the "human" half-space, and a third came within 0.04 margin units. None of the trajectories plateaued on the AI side, suggesting further iterations could flip the remaining targets.

Significance and Claims
The paper claims three primary contributions:

1. Reproducibility Efficiency: It demonstrates that with open data and an agentic research harness, full reproduction and extension of a complex NLP study can be achieved in hours rather than weeks, shifting the bottleneck from compute to openness.
2. LLM Capability: It provides evidence that frontier LLMs, when provided with a single style demonstration, can already mimic a specific author's style more effectively than humans editing a draft, closing the majority of the stylistic gap.
3. The Arms Race Reality: It frames AI-text detection as an active, asymmetric arms race. While detectors currently maintain high accuracy, the nature of the signal differs by model (length vs. style). Most significantly, the study shows that a frontier LLM agent, given only inference-time feedback from a detector, can efficiently lower its own detection probability to the point of crossing into human classification within a short number of iterations.

The authors conclude that the current snapshot favors the agent, noting that the adversarial loop has not yet exhausted the optimization potential and that direct weight updates (fine-tuning) would likely be even more effective. All code, data, and trained detectors are released to serve as a reference point for future iterations of this race.