📊 statistics

Token-Level Likelihood-Array Regression for Membership Inference and AI-Generated Text Detection

This paper proposes Likelihood-Array Regression (LAR), a novel method that organizes token-level probabilities across nested context windows into structured arrays to significantly improve membership inference and AI-generated text detection by capturing nuanced information about context scale and token position that traditional likelihood-based baselines miss.

Original authors: Jiajun Sun, Zhanrui Cai

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jiajun Sun, Zhanrui Cai

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the digital age, language models have become so proficient at mimicking human writing that distinguishing between a sentence written by a person and one generated by a machine has grown increasingly difficult. This challenge sits at the heart of two critical questions facing researchers and society today. The first is a matter of privacy and copyright: can we tell if a specific piece of text was used to train a model in the first place? This is vital for publishers and authors who want to know if their work was ingested by artificial intelligence without permission. The second question is about origin: was a given text created by a human or by an algorithm? This distinction is essential for maintaining trust in journalism, academic work, and online discourse. To answer these questions, scientists have long relied on a method that looks at how likely a model is to predict the next word in a sentence. If a model is very confident about the next word, it suggests the text might be familiar to it or generated by it. However, traditional methods have treated this confidence as a single, static number, often ignoring how the model's certainty changes as it reads more or less of the surrounding sentence.

A new study by researchers Jiajun Sun and Zhanrui Cai proposes a more nuanced way to look at this data, treating the confidence of a language model not as a single point, but as a rich, structured landscape. Instead of asking the model to judge a word based on the entire sentence that came before it, the researchers asked the model to judge the same word repeatedly, each time with a different amount of context. They started with just the immediate previous word, then added one more, then two, and so on, all the way up to the full sentence. This process created a detailed map of how the model's confidence shifts as the amount of available information grows. By organizing these thousands of confidence scores into a structured array, the researchers could see patterns that were invisible when looking at the full sentence alone. They then used a statistical technique to learn which parts of this map—whether short contexts, long contexts, or specific positions in the text—were the most reliable indicators of whether a text was human-written, machine-generated, or part of the model's training data.

The results of this approach were strikingly effective. When tested on membership inference, which asks if a text was used to train a model, the new method achieved a success rate between 91.4% and 98.3% across five different language models. This was a significant improvement over existing methods, which typically hovered between 50% and 87%. For detecting AI-generated text, the method performed even better, reaching success rates between 98.5% and 99.6%. The study revealed that the most valuable information often came from the shorter contexts. Contrary to the assumption that a model needs the full sentence to make a judgment, the researchers found that the model's reaction to a word based on just a few preceding words contained unique signals that the full-sentence view missed. Furthermore, the study showed that for membership inference, looking at how the model's confidence changed between different context lengths provided extra clues, allowing the system to detect subtle patterns of repetition or memorization that simpler methods overlooked.

The researchers also explored how much data was needed to make these detections accurate. They found that even with a very small number of labeled examples, the new method outperformed traditional approaches. With as few as one hundred labeled texts, the system could already distinguish between human and machine writing with high reliability, and its accuracy continued to climb as more data was added. This suggests that the method is efficient and does not require massive datasets to be effective. The study also demonstrated that the approach works well even when the model used to analyze the text is different from the model that generated it. This is a crucial finding, as it means the technique can be applied broadly without needing to know the exact identity of the AI that created the content.

By moving away from single-number summaries and embracing the full complexity of how a language model processes context, this research offers a powerful new tool for auditing artificial intelligence. It confirms that the path to understanding AI behavior lies not in simplifying the data, but in organizing it in a way that respects its structure. The work provides a clear, data-driven path forward for protecting intellectual property and verifying the authenticity of text in an era where the line between human and machine creation is increasingly blurred. The findings suggest that by paying attention to the subtle shifts in a model's confidence as it reads, we can uncover the hidden fingerprints of both the training data and the generation process, offering a robust defense against misuse and a clearer view of the digital landscape.

Technical Summary: Token-Level Likelihood-Array Regression for Membership Inference and AI-Generated Text Detection

Problem Statement
The paper addresses two critical auditing tasks for language models: membership inference (determining if a text was part of a model's training data) and AI-generated text detection (distinguishing model-generated text from human writing). While both tasks can be framed as binary classification problems based on conditional token probabilities, existing likelihood-based methods suffer from significant information loss. Current approaches typically compress token-level probabilities into a few scalar summaries (e.g., average loss, Min-K%, or perplexity), often relying solely on probabilities conditioned on the full preceding context ( $\ell_{full}(t)$ ). This aggregation discards the joint variation of likelihood information across different context lengths and token positions. Furthermore, supervised detectors trained on labeled corpora often require extensive fine-tuning and additional labeled data to adapt to new models or domains. The authors pose the statistical question of how to represent and model likelihood information that varies jointly with context length and token position without imposing rigid structural assumptions.

Methodology: Likelihood-Array Regression (LAR)
The authors propose Token-Level Likelihood-Array Regression (LAR), a framework that treats the conditional token probabilities from a fixed scoring language model as a structured, variable-length array rather than a scalar summary.

Likelihood Array Construction:
- For each target token $x_t$ in a text, the scoring model evaluates the log-probability under a sequence of nested left-context windows, ranging from the full preceding sequence ( $x_1, \dots, x_{t-1}$ ) down to a single token ( $x_{t-1}$ ).
- This creates a "target-token path" for each token, recording how the model's assessment changes as the context scale decreases.
- These paths form a triangular likelihood array $L(X) = \{(s, t, \ell(s, t))\}$ , where $s$ is the start of the context and $t$ is the target token.
Alignment and Feature Engineering:
- To handle variable text lengths, the authors normalize the context scale ( $u$ ) and relative token position ( $v$ ) to a common domain $[0, 1] \times [0, 1]$ .
- Cells are aligned via linear interpolation onto a fixed grid of context scales.
- Each aligned cell is augmented with a 5-dimensional channel vector $h$ $h$ :
  - Observed-token log probability ( $\ell$ ).
  - Context contrast ( $\delta$ ): The log-ratio of the token probability with context versus without context (BOS).
  - Standardized log probability ( $z$ ) and standardized context contrast ( $z_\delta$ ): Normalized against the model's predictive vocabulary distribution.
  - Local slope ( $\dot{z}$ ): The rate of change in standardized predictability as context increases.
Regression Models:
- LAR-1 (First-Order): Models the class log-odds as an additive aggregation of contributions from individual aligned cells. It learns a function $b(u, v, h)$ mapping context scale, position, and features to the log-odds, effectively learning how likelihood patterns contribute differently depending on their location in the document and context length.
- LAR-2 (Second-Order): Augments LAR-1 with within-path quadratic features. It captures interactions between pairs of evaluations of the same target token across different context lengths. This allows the model to detect how the change in likelihood (e.g., a low likelihood under short context that rises under long context) informs the label.
Estimation:
- The infinite-dimensional functional models are approximated using finite basis expansions (tensor products of splines for context/position and random trigonometric functions for channels).
- The second-order terms are estimated using random squared projections to reduce dimensionality while preserving the within-path structure.
- The final estimators are fitted via ridge logistic regression on document-level features derived from these expansions.

Key Contributions

Novel Representation: The paper introduces the likelihood-array representation, which preserves the joint variation of likelihoods across context scales and token positions, moving beyond the scalar summaries used in prior work.
Theoretical Bounds: For the within-path quadratic model (LAR-2), the authors establish matching minimax lower and upper prediction bounds of the rate $\{\log(en)/n\}^{\nu/(\nu+1)}$ under product-form covariance decay. They characterize errors arising from finite-dimensional approximation and random projections, showing conditions under which an oracle spectral sieve attains the minimax rate.
Empirical Superiority: The framework is shown to substantially outperform existing likelihood-based baselines and supervised detectors across multiple scoring models and domains.
Insight into Information Sources: The analysis reveals that shorter-context likelihoods contain significant information beyond conventional full-context probabilities and that second-order features provide substantial gains specifically for membership inference.

Results
The authors evaluated LAR on two benchmarks:

WikiMIA (Membership Inference): Using 1,650 texts and five scoring models (Pythia, LLaMA-2, Falcon).
- LAR-1 achieved mean AUCs of 0.914–0.960.
- LAR-2 improved this to 0.953–0.983, significantly outperforming the best existing baseline (ReCaLL, max AUC 0.871) and a fine-tuned supervised method (Fine-tuned Score Deviation, max AUC 0.90).
- Shorter-context cells alone yielded AUCs of ~0.907, compared to ~0.802 for full-context cells only.
AI-Generated Text Detection: Using 1,200 texts from four domains (fiction, news, essays, reviews) generated by GPT-4o and scored by five open-weight models.
- LAR-1 achieved mean AUCs of 0.985–0.996.
- LAR-2 achieved 0.987–0.995.
- These results surpassed all baselines, including the supervised detector StatDetectLLM (AUC 0.980) and RADAR (AUC 0.839).
- In this task, the gain from second-order features was minimal, suggesting that first-order functional patterns capture most of the separation signal.

Significance and Claims
The paper claims that LAR provides a statistically principled solution to the "broader statistical question" of how to represent likelihood information varying with context and position. By avoiding the aggregation of token-level data into prespecified scores, LAR extracts richer signals, particularly from shorter context windows which are often ignored. Theoretically, the work provides the first minimax analysis for this specific class of within-path quadratic models, linking prediction error to AUC convergence. Practically, the method offers a robust, non-fine-tuned alternative to supervised detectors, requiring only a fixed scoring model and a labeled dataset for the regression stage, while achieving state-of-the-art performance in both privacy auditing and AI detection. The authors note that while the method is computationally more demanding than full-context scoring, the performance gains justify the cost, and future work could explore adaptive context window selection to reduce computational overhead.

More like this