Technical Summary: Black-Box Detection of LLM-Generated Text Using Generalized Jensen-Shannon Divergence

Problem Statement

The rapid advancement of Large Language Models (LLMs) has blurred the distinction between human-written and machine-generated text, raising concerns regarding misinformation, plagiarism, fraud, and the degradation of training data quality through recursive feedback loops. Existing detection methods face significant limitations in practical, black-box scenarios:

• Classifier-based detectors require costly retraining when domains or generators shift and rely on high-quality labeled data.
• Global-statistic methods (e.g., likelihood, rank, entropy) are often unstable under calibration mismatches and domain shifts.
• Distributional-statistic methods (e.g., DetectGPT, DNA-GPT) typically require per-instance regeneration or perturbation to estimate local divergence, leading to high computational costs and latency that are unrealistic for high-throughput applications.

Furthermore, black-box constraints exacerbate issues where the proxy model used for scoring may mismatch the unknown source model. The paper addresses the need for a detector that avoids retraining and per-instance regeneration while remaining robust under distribution shifts and proxy-model mismatches.

Methodology: SurpMark

The authors propose SurpMark, a reference-based, black-box detector that frames attribution as a likelihood-free hypothesis test. The framework operates in two phases:

1. Offline Reference Construction

Using existing human and machine corpora, the system builds fixed reference transition matrices:

• Surprisal Estimation: A fixed proxy Language Model (LM) computes token-level surprisals ($s_t = -\log p_\theta(x_t | x_{<t})$) for reference texts.
• Discretization: Continuous surprisal values are quantized into $k$ interpretable states (e.g., "Predictable," "Surprising") using a shared $k$-means quantizer. This converts the sequence into a discrete state sequence $\{a_t\}$.
• Transition Modeling: The state sequences are modeled as first-order Markov chains. Empirical transition probability matrices ($\hat{M}_{human}$ and $\hat{M}_{machine}$) are estimated by counting state transitions in the reference corpora.

2. Online Detection

For a test text passage:

• Feature Extraction: The same proxy LM computes surprisals, which are discretized using the shared quantizer to form a state sequence.
• Transition Estimation: An empirical transition matrix ($\hat{M}_{test}$) is computed for the test text.
• Scoring: The detector calculates a Generalized Jensen-Shannon (GJS) divergence score. Specifically, it computes the difference between the GJS distance to the machine reference and the GJS distance to the human reference:
  $$\Delta GJS_n = GJS(\hat{M}_{machine}, \hat{M}_{test}, \alpha) - GJS(\hat{M}_{human}, \hat{M}_{test}, \alpha)$$
  where $\alpha$ is the ratio of reference length to test length.
• Decision: The text is classified as machine-generated if $\Delta GJS_n \leq \tau$ (where $\tau$ is a threshold, typically 0), and human-written otherwise.

Theoretical Justification

The paper provides a theoretical analysis under an idealized first-order Markov approximation of the discretized surprisal states:

• Discretization-Estimation Tradeoff: The authors derive a scaling law for the number of bins $k$. Balancing discretization error ($O(1/k)$) and statistical estimation error ($O(k^{3/2}/\sqrt{N})$), the optimal bin count scales as $k^* = \Theta(N^{1/5})$, where $N$ is the total length of the reference data.
• Statistic Validity: The $\Delta GJS_n$ statistic is shown to be equivalent to the normalized log-likelihood ratio between the two hypotheses (human vs. machine). The authors prove its asymptotic normality, providing a principled basis for the decision rule.
• First-Order Sufficiency: Empirical and theoretical analysis (based on Gray's approximation theory) indicates that the conditional mutual information gained by moving from a first-order to a second-order Markov model is negligible (sub-percent perplexity reduction), justifying the use of first-order models for data efficiency.

Key Contributions

1. SurpMark Framework: A reference-based detector that summarizes text via token surprisal dynamics (state transitions) rather than absolute likelihoods, avoiding the need for per-instance regeneration.
2. Theoretical Analysis: Derivation of design guidelines for discretization bin scaling ($k \propto N^{1/5}$) and a principled justification for the GJS-based test statistic as a log-likelihood ratio.
3. Comprehensive Evaluation: Extensive experiments demonstrating robustness across diverse datasets, source models (open and closed-source), and scenarios (mixed domains, paraphrasing, adversarial attacks).

Experimental Results

SurpMark was evaluated against 13 baselines (including DetectGPT, Lastde++, R-Detect, and Binoculars) across multiple datasets (XSum, WritingPrompts, SQuAD, WMT19, HC3, DetectRL) and models (9 open-source, 3 closed-source).

• Performance: SurpMark consistently matches or surpasses baselines. It achieved the highest average AUROC on 6 of 9 open-source models and ranked first on all 3 closed-source models (including GPT-5-Chat), where marginal statistics often fail.
• Efficiency: Unlike distributional methods requiring per-input regeneration (e.g., DetectGPT, Lastde++), SurpMark scores texts in a single pass. It achieves higher throughput than baselines after amortizing the one-time preprocessing cost, making it suitable for real-time deployment.
• Robustness:
  • Proxy Mismatch: SurpMark remains stable even when the proxy LM differs significantly from the source generator, whereas many baselines degrade.
  • Domain Shift: The method maintains high performance in mixed-domain and mixed-generator settings (DetectRL benchmark).
  • Adversarial Attacks: SurpMark demonstrates strong resilience against paraphrasing attacks (polishing, back-translation, DIPPER) and prompt-engineered adversarial attacks designed to mimic human style or evade detection.
  • OOD Generalization: The detector performs well when reference texts come from a different domain than the test set, indicating it captures transferable signals rather than domain-specific artifacts.

Significance and Claims

The paper claims that SurpMark offers a robust, efficient, and theoretically grounded solution for black-box LLM text detection. Its significance lies in:

• Decoupling from Source Models: By relying on the dynamics of token surprisals relative to fixed references, it avoids the brittleness of classifiers trained on specific generators.
• Computational Efficiency: It eliminates the latency and cost associated with per-instance regeneration, enabling scalable deployment.
• Handling Advanced Models: The method specifically addresses the challenge of detecting text from strong, closed-source models (like GPT-5) where marginal surprisal distributions converge with human text, but transitional dynamics remain distinct.

The authors modestly note limitations, acknowledging that the first-order Markov approximation is an idealization for the discretized state space and that pretraining exposure of human texts in the proxy model could theoretically increase false positives, though experiments suggest this effect is minimal. They emphasize that the detector provides a probabilistic signal and should not be used as the sole basis for high-stakes decisions without human review.