MGTEVAL: An Interactive Platform for Systemtic Evaluation of Machine-Generated Text Detectors

1. Problem Statement

The rapid advancement of Large Language Models (LLMs) has made generating human-like text at scale trivial, raising concerns about misinformation, phishing, and academic dishonesty. While numerous Machine-Generated Text (MGT) detectors have been developed to distinguish AI content from human writing, the field suffers from a lack of standardized evaluation.

• Fragmentation: Existing evaluations are scattered across different datasets, preprocessing steps, attack methods, and metrics.
• Reproducibility Issues: Results are difficult to compare or reproduce due to inconsistent experimental settings.
• Limited Scope: Current frameworks often lack integrated support for dataset construction, robustness testing (via attacks), detector training, and comprehensive performance reporting within a single system.

2. Methodology: The MGTEVAL Framework

MGTEVAL is a unified, extensible platform designed to standardize the entire lifecycle of MGT detector evaluation. It employs a modular, registry-based design where all components (detectors, attacks, datasets) are managed through a configuration-driven interface. The framework consists of four core components:

A. Dataset Building

• Function: Standardizes heterogeneous corpora into a unified binary detection format.
• Capabilities:
  • Supports loading existing datasets (e.g., HC3, SemEval 2024, M4) and custom datasets.
  • Enables custom dataset construction by pairing human-written texts with machine-generated counterparts produced by configurable LLM backends.
  • Allows full configuration of generation parameters (prompts, temperature, top-k/p, seeds) to synthesize specific MGT samples.
  • Automatically labels data (0 for human, 1 for machine) and annotates metadata.

B. Dataset Attack (Robustness Evaluation)

• Function: Generates adversarial examples to stress-test detector robustness.
• Mechanism:
  • Applies 12 common text attack methods (e.g., humanization, deletion, paraphrasing, insertion) specifically to machine-generated samples (label=1), leaving human samples unchanged to preserve the reference distribution.
  • Generates paired variants with provenance metadata, enabling traceable robustness analysis.

C. Detector Training

• Function: Provides a unified pipeline for training both metric-based and model-based detectors.
• Metric-based Detectors: Treats calibration as the training step. It fits a logistic regression model to map raw detector scores into probabilities, ensuring fair thresholding across heterogeneous detectors.
• Model-based Detectors: Performs end-to-end supervised optimization (e.g., fine-tuning BERT/RoBERTa) on the unified binary dataset.
• Pre-trained Models: Includes 26 existing detectors trained on standardized datasets, released as "off-the-shelf" models to save user effort.

D. Performance Evaluation

• Function: Automates the evaluation process across three dimensions:
  1. Effectiveness: Reports Accuracy (ACC), F1, Precision, Recall, AUROC, and AUPR.
  2. Robustness: Uses Attack Success Rate (ASR) to quantify how often attacks flip correct decisions.
  3. Efficiency: Measures runtime metrics including latency (ms/sample), throughput, and GPU peak memory usage.
• Thresholding: Uses fixed thresholds determined during training/calibration to ensure fair comparison between clean and attacked settings, avoiding re-tuning on adversarial data.

E. User Interface

• Dual Interface: Offers both a Command-Line Interface (CLI) and a Web-based UI.
• Features: The WebUI provides a task-oriented control panel mirroring the backend pipeline, allowing non-experts to configure experiments, monitor real-time logs, and visualize results without code rewriting.

3. Key Contributions

1. Unified Evaluation Pipeline: MGTEVAL is the first framework to integrate dataset construction, attack generation, detector training, and multi-metric evaluation into a single, reproducible system.
2. Comprehensive Benchmarking Suite:
   • Supports 26 existing MGT detectors (both metric-based and model-based).
   • Includes 12 distinct attack methods for robustness testing.
   • Reports ≥5 built-in metrics covering effectiveness, robustness, and efficiency.
3. User-Friendly Extensibility: The registry-based architecture allows users to seamlessly add new detectors or attacks without modifying the core framework. It supports custom datasets and attacks via both CLI and WebUI.

4. Experimental Results

The authors evaluated 26 detectors on a balanced dataset (2,000 samples: 1,000 human from SemEval 2024, 1,000 machine from Qwen3-4B).

• Top Performer: Longformer achieved the best overall performance with 99.50% Accuracy, 99.50% F1, and 99.99% AUROC. It also showed superior robustness at low False Positive Rates (TPR@FPR=0.01 was 100%).
• Efficiency vs. Effectiveness Trade-offs:
  • PECoLA was the fastest (6.86 ms/sample).
  • GREATER offered the best balance of high performance (97.50% Acc) and extreme efficiency (0.14 GiB GPU peak memory), significantly outperforming others in resource usage.
  • RAiDAr achieved high accuracy (98.00%) but incurred massive computational costs (427.40 ms/sample, 14.99 GiB GPU).
• Metric Discrepancies: The study highlighted that threshold-free metrics (like AUROC) can be misleading. For instance, TOCSIN had a high AUROC (95.03%) but poor Accuracy (59.00%) and zero TPR at strict FPRs, indicating sensitivity to threshold selection.
• Failure Cases: Detectors like RADAR and Entropy showed severe performance degradation, underscoring the need for systematic benchmarking.

5. Significance and Impact

• Standardization: MGTEVAL addresses the critical lack of a "gold standard" for MGT detection, enabling fair, apples-to-apples comparisons across the research community.
• Holistic Assessment: By integrating efficiency and robustness metrics alongside accuracy, it prevents the deployment of detectors that are accurate but too slow or easily bypassed.
• Accessibility: The WebUI lowers the barrier to entry, allowing researchers and practitioners to evaluate detectors without deep coding expertise.
• Future-Proofing: The modular design ensures the platform can evolve to support new LLMs, multilingual benchmarks, and emerging attack vectors.

Limitations: The full pipeline can be computationally expensive; some detector-specific hyperparameters may be simplified under the unified protocol; and the current version focuses primarily on binary detection and text-level attacks, not yet covering fine-grained source attribution or cross-domain scenarios.

Ethical Considerations: The authors emphasize that the attack generation module is for defensive stress-testing. They warn against the dual-use risk of using these tools to evade detection and recommend human oversight when deploying detectors in high-stakes decisions.