💬 NLP

On the Salience of Low-Probability Tokens for AI-Generated Text Detection: A Multiscale Uncertainty Perspective

This paper introduces "Uncertainty," a multiscale detection method that improves AI-generated text identification by focusing on low-probability tokens to mitigate boilerplate dominance and reduce brittleness through local averaging and global Rényi entropy analysis, achieving superior effectiveness and robustness across diverse datasets and models.

Original authors: Yikai Guo, Bin Wang, Xilai Fan, Wenjun Ke, Haoran Luo

Published 2026-06-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yikai Guo, Bin Wang, Xilai Fan, Wenjun Ke, Haoran Luo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "AI vs. Human" Blur

Imagine a world where AI can write stories, emails, and news articles that sound almost exactly like they were written by a human. This is great for creativity, but it creates a problem: How do we tell the difference?

If we can't tell them apart, bad actors could spread misinformation, students could cheat, and the internet could get flooded with fake content.

Current tools try to spot AI text by looking at the "statistical fingerprint" of the words. However, the paper argues that these tools have two major flaws:

The "Boilerplate" Problem: AI and humans both use common phrases like "In conclusion," "The results show," or "We propose." Current detectors get distracted by these boring, common words. It's like trying to find a thief in a crowd by looking at everyone's shoes; since everyone wears shoes, you can't tell who the thief is. The common words drown out the unique clues.
The "Brittle" Problem: Current tools often rely on a single number (a score) to make a decision. If someone slightly rephrases a sentence or changes the AI's settings, that single number can jump around wildly, causing the detector to flip-flop between "Human" and "AI." It's like a scale that tips over if you put a feather on it.

The Solution: "Uncertainty"

The authors propose a new method called Uncertainty (and a stronger version called Uncertainty++). Instead of looking at the whole text equally, they focus on two specific things: the rare words and the shape of the choices.

Analogy 1: The "Surprise Party" (Low-Probability Tokens)

Imagine you are at a party.

High-probability tokens (Common words): Everyone says "Hello," "How are you?" or "Nice weather." These are predictable. Both humans and AI say these all the time.
Low-probability tokens (Rare words): Suddenly, someone says something weird and unexpected, like "The toaster is singing opera."

The paper's main discovery is that AI is much more predictable than humans when it comes to these "surprise" moments.

When a human writes, they might take a risk and choose a unique, low-probability word.
When an AI writes, even if it tries to be creative, it tends to stick to "safe" choices. If it does pick a rare word, it's usually because the math forced it there, making the word feel "off" or statistically different from how a human would choose it.

The Method: The new detector ignores the boring "Hello" and "How are you" parts. It zooms in only on the weird, rare words (the bottom 15% of choices). By ignoring the noise, it finds the signal much more clearly.

Analogy 2: The "Weather Forecast" (Global Uncertainty)

Current detectors look at the specific word chosen (the "point estimate").

Old way: "The AI picked the word 'cat'. That's a 90% chance. Score: High."
New way: The paper asks, "What did the AI think about all the other options?"

Imagine a weather forecast.

Old way: The forecaster says, "It will rain." (One single prediction). If you change the forecast slightly, the whole decision changes.
New way (Rényi Entropy): The forecaster looks at the entire sky. "There's a 40% chance of rain, 30% of sun, 20% of clouds, and 10% of snow." This "shape" of the forecast is much harder to trick. Even if you change the specific word "rain" to "drizzle," the overall shape of the sky (the uncertainty) stays stable.

The paper uses a mathematical tool called Rényi entropy to measure this "shape" of the AI's brain at every moment. This makes the detector robust against people trying to trick it by rephrasing or changing settings.

How It Works Together

The Uncertainty detector combines these two ideas:

Local Check: It looks at the rare, surprising words to see if they feel "off" compared to human writing.
Global Check: It looks at the "shape" of the AI's choices to ensure the decision isn't easily broken by small edits.

They also created Uncertainty++, which is like running the test multiple times with slightly different conditions to get a more stable average, ensuring the result doesn't change just because of a tiny glitch.

The Results

The authors tested this on 16 different AI models (including the newest ones) and 7 different types of writing (from news to Reddit comments).

Better Accuracy: It beat all the previous best methods.
Harder to Trick: It stayed accurate even when the text was rewritten or the AI's settings were changed.
Fast: It works efficiently without needing massive computing power.

Summary

Think of old detectors as a security guard checking everyone's ID card (the common words). The new Uncertainty detector is a detective who ignores the ID cards and instead watches how people react when asked a tricky question (the rare words) and how nervous they seem overall (the shape of their choices). This makes it much harder for an AI to sneak past.

Technical Summary: On the Salience of Low-Probability Tokens for AI-Generated Text Detection

Problem Statement

As Large Language Models (LLMs) evolve, distinguishing AI-generated text from human writing has become increasingly difficult, posing risks such as misinformation, academic misuse, and corpus contamination. While statistical detectors offer efficiency and generalization compared to watermarking or fine-tuning approaches, they face two critical limitations:

Boilerplate Dominance: Aggregating token-level probabilities across an entire sequence often allows high-probability, routine "boilerplate" tokens (common to both human and AI writing) to overwhelm the discriminative signals found in more informative tokens.
Brittle Point Estimates: Relying on a single probability score (a point estimate) for decision-making discards the shape of the underlying conditional distribution. This makes detectors sensitive to surface-level perturbations, such as changes in decoding strategies or paraphrasing, which can drastically shift scores and flip predictions.

Methodology

The authors propose Uncertainty and Uncertainty++, a multiscale uncertainty framework designed to address these limitations by focusing on low-probability tokens and capturing distributional shape.

Core Hypothesis

The method is grounded in the Low-Probability Discriminability Hypothesis: discriminative signals between human and AI text are substantially larger at low-probability tokens than at high-probability ones. Empirical analysis shows that while high-probability tokens often share similar likelihoods across both classes, low-probability tokens exhibit a significantly larger gap in log-probability and probability ratios.

The Framework

The framework operates in two complementary scales:

Local Uncertainty (Percentile-Based):
- Instead of averaging over all tokens, the method ranks token positions by their observed log-probabilities and selects the bottom $\rho$ fraction (the "tail").
- It computes the mean log-probability of these low-probability tokens ( $z_{local}$ ).
- To improve robustness, Uncertainty++ employs conditional independent sampling. It samples tokens from the model's own conditional distribution (conditioned on the original prefix) to estimate the expected discrepancy. The local signal is normalized by this expectation ( $D^*_\rho$ ), effectively calibrating the score against the model's distributional baseline.
Global Uncertainty (Entropy-Based):
- To capture the shape of the conditional distribution rather than just a single token realization, the method computes the Rényi entropy ( $H_\alpha$ ) of the conditional distribution at each position.
- This entropy is averaged over the same low-probability token positions ( $z_{global}$ ).
- Theoretical analysis (Proposition 3.4) demonstrates that Rényi entropy is more robust to multiplicative perturbations (e.g., paraphrasing) than raw log-probability, as its change remains bounded while log-probability changes linearly with the perturbation factor.

Unified Detection Score

The final detection score fuses these two signals:

Uncertainty: A weighted sum of the raw local log-probability and the global entropy.
Uncertainty++: A weighted sum of the normalized local discrepancy ( $D^*_\rho$ ) and the global entropy. This variant provides a more stable estimation by accounting for the model's inherent uncertainty.

Key Contributions

Empirical Validation of Low-Probability Hypothesis: The authors identify and validate that low-probability tokens carry significantly stronger discriminative evidence than high-probability boilerplate, with the signal gap being up to 3.5 times larger in low-probability regions.
Multiscale Uncertainty Estimator: They propose a novel detector that fuses local tail log-probability with global Rényi entropy, addressing both boilerplate dominance and the brittleness of point estimates.
Robustness via Conditional Sampling: The introduction of Uncertainty++ utilizes conditional independent sampling to normalize local signals, enhancing stability against decoding variations and paraphrasing.

Experimental Results

The methods were evaluated across seven datasets (including XSum, SQuAD, WritingPrompts, Reddit, FiQA, Wiki-csai, and arXiv) and sixteen LLMs (ranging from GPT-2 to GPT-5 and Gemini-3).

Performance: In the black-box setting, Uncertainty achieved an average AUROC of 88.74%, outperforming the strongest probability-based baseline (Lastde) by 2.25 points. Uncertainty++ achieved an average AUROC of 93.24%, surpassing the strongest sampling-based baseline (Lastde++) by 1.73 points.
Generalization: The methods demonstrated strong cross-domain generalization, maintaining high performance on specialized domains like finance (FiQA) and academic abstracts (arXiv), though performance naturally dipped on highly specialized content.
Robustness: The detectors showed superior resilience to decoding strategy variations (temperature, top-p, frequency penalty) and paraphrasing attacks compared to baselines. For instance, under paraphrasing attacks, Uncertainty++ maintained an average AUROC of 81.65%, outperforming Lastde++ by 2.67 points.
Ablation: Removing the low-probability selection module caused a significant performance drop (-6.74 AUROC), confirming the importance of focusing on the tail. Removing the entropy term caused a minor drop (-0.89), suggesting it acts as a stabilizer.

Significance and Claims

The paper claims that Uncertainty and Uncertainty++ provide a more reliable statistical signal for identifying AI-generated text by shifting focus from ubiquitous high-probability tokens to informative low-probability regions. By capturing both local and global distributional properties, the approach offers a robust solution that is less susceptible to the "boilerplate dominance" and "brittle point estimates" that plague existing statistical detectors.

The authors emphasize that while their methods improve detection, they should be used as decision-support tools alongside human review. They explicitly caution against treating detector outputs as sole evidence in high-stakes decisions, advocating for the reporting of uncertainty when evidence is weak and the calibration of thresholds based on specific application risks. Future work is proposed to further interpolate uncertainty for sparse evidence and strengthen domain adaptation for specialized texts.