Team DACTYL at PAN 2026: Bayesian Data Mixing and Empirical X-risk Minimization for AI-text Detection

Technical Summary: Bayesian Data Mixing and Empirical X-risk Minimization for AI-text Detection

Problem Statement
Current AI-generated (AIG) text detection classifiers often exhibit strong in-distribution (ID) performance but fail to generalize to out-of-distribution (OOD) texts. The authors hypothesize that this failure stems from overfitting to dataset-specific spurious correlations and superficial features, which allow even small models (e.g., BERT-tiny) to achieve deceptively high scores on specific benchmarks. Furthermore, standard training via Empirical Risk Minimization (ERM) tends to encourage "shortcut learning," while naive dataset combination can reinforce dependency on these spurious correlations. Additionally, existing classifiers often lack calibrated confidence scores, and techniques like hard mining fail to distinguish between genuinely difficult samples and noisy or corrupted data.

Methodology
The Team DACTYL approach addresses these issues through a pipeline involving Bayesian data filtering, empirical X-risk minimization (EXM), and post-hoc multicalibration.

1. Dataset Construction (DACTYLv2):
   The team constructed an expanded dataset, DACTYLv2, by generating AIG texts via one-shot prompting across six domains (blogs, web articles, Medium, movie scripts, NBER abstracts, Yelp reviews) using various LLMs (including Claude, GPT, GLM, MiniMax, Mercury, and Kimi). This was combined with external datasets (LLMTrace, Personagen, Stack Exchange, MAGA-Bench, DetectRL) to create a diverse training pool and prevent overfitting to DACTYL-specific artifacts.

2. Bayesian Data Filtering:
   To curate a high-quality consolidated training set, the authors trained five weak Bayesian BERT-tiny classifiers (BERT-tiny with a Bayesian linear head) on samples from different source datasets.

   • Mechanism: These models output a mean score ($\mu$) and standard deviation ($\sigma$) for each input, representing prediction confidence and uncertainty.
   • Filtering Logic: Texts were evaluated against four quadrants: correct/uncertain, correct/certain, incorrect/uncertain, and incorrect/certain.
   • Selection Criteria: Samples were "kept" if they were correctly classified or if they were incorrectly classified but uncertain (indicating a hard-to-learn case). Samples were "rejected" if the classifier was incorrect and certain, as these likely represent outliers, mislabeled data, or spurious correlations. An ensemble voting mechanism (requiring at least three "keep" votes and no more than one "reject" vote) was used to finalize the training set.

3. Model Training with Empirical X-risk Minimization (EXM):
   Instead of standard ERM (cross-entropy), the team fine-tuned DeBERTa-V3-large and ModernBERT-large models on the filtered dataset using two-way partial AUROC optimization (via the LibAUC library). This approach is designed to improve generalizability, particularly for larger models.

4. Multicalibration (MCGrad):
   For the ModernBERT-large model, a post-hoc calibration step was applied using MCGrad. This model takes the transformed score from ModernBERT-large and the uncertainty estimates ($\sigma$) from the three Bayesian BERT-tiny classifiers (trained on DACTYL, LLMTrace, and MAGA-Bench) as inputs. This step aims to align the predicted probabilities with true labels, optimizing for metrics like the Brier score.

Key Contributions

• DACTYLv2: A new, expanded dataset for AIG detection incorporating diverse domains and prompt styles.
• Bayesian Filtering Framework: A novel method using Bayesian neural networks to identify and remove "incorrect and certain" samples (potential outliers/spurious correlations) from training data, effectively blending multiple datasets.
• EXM Implementation: The application of empirical X-risk minimization (partial AUROC) for fine-tuning large language models in the context of AIG detection.
• Uncertainty-Driven Calibration: A system that leverages the uncertainty estimates of weak Bayesian models to calibrate the predictions of a strong, non-Bayesian classifier.

Results
The team trained and evaluated three classifiers: DeBERTa-V3-large, ModernBERT-large, and the MCGrad-calibrated ModernBERT-large.

• Performance: The MCGrad model achieved the highest mean score of 0.974 on the PAN 2026 test set across five metrics (AUROC, $F_1$, C@1, Brier score, and $F_{0.5u}$), ranking second on the leaderboard.
• Comparison: The uncalibrated ModernBERT-large achieved a mean score of 0.96, while DeBERTa-V3-large achieved 0.882.
• Generalization: Both large models demonstrated strong OOD performance, with AUROC scores exceeding 0.93 on several external test sets (e.g., DetectRL, Dolly-Cosmopedia) despite not being trained on them.
• Calibration Impact: The MCGrad system improved scores on the PAN 2025 and 2026 test sets (including the Brier score, where it ranked first among competitors) but showed mixed results on the ELOQUENT test set, suggesting sensitivity to specific distribution shifts.

Significance and Claims
The paper claims that careful dataset curation using Bayesian uncertainty filtering is a critical factor in achieving strong OOD performance, rather than simply aggregating large volumes of data. The authors highlight that high discrimination capacity (high AUROC) does not guarantee calibrated scores, emphasizing the necessity of evaluating across multiple metrics. The work demonstrates that combining Bayesian data selection with EXM and multicalibration can produce robust detectors that outperform previous state-of-the-art systems (specifically beating the PAN 2025 winning system in overall score). The authors note that their Bayesian hyperparameters were manually tuned and suggest that future optimization via proxy uncertainty metrics could further improve results.