💬 NLP

Measuring AI "Slop" in Text

This paper addresses the lack of a standard definition for AI "slop" by developing a taxonomy and interpretable dimensions through expert interviews, demonstrating that while binary judgments are subjective, they correlate with latent factors like coherence and relevance to enable better evaluation of AI-generated text.

Original authors: Chantal Shaib, Tuhin Chakrabarty, Diego Garcia-Olano, Byron C. Wallace

Published 2026-01-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Chantal Shaib, Tuhin Chakrabarty, Diego Garcia-Olano, Byron C. Wallace

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet is a giant, bustling marketplace. For a long time, vendors (humans) sold fresh, handcrafted goods. But recently, a flood of cheap, mass-produced items has arrived. They look okay from a distance, but up close, they feel flimsy, repetitive, and hollow. In the world of AI, people have started calling this low-quality, AI-generated junk "slop."

This paper is like a team of experts trying to build a quality control manual for this slop. They want to answer two big questions: What exactly makes text feel like "slop"? and Can we build a machine to spot it automatically?

Here is the breakdown of their findings, using simple analogies:

1. Defining the "Slop" (The Taxonomy)

The researchers didn't just guess what slop is. They interviewed 19 experts—writers, journalists, philosophers, and AI scientists—to get a clear definition. They realized "slop" isn't just one thing; it's a mix of three main problems, like a bad meal that is too salty, too bland, and served on a dirty plate.

They organized these problems into a checklist called a Taxonomy:

Information Utility (The "Empty Bowl"):
- Density: The text is full of words but has very little actual meat. It's like a soup that is 90% water and 10% broth.
- Relevance: The text talks about things that don't matter to the question asked. It's like ordering a burger and getting a lecture on the history of cows.
Information Quality (The "Spoiled Ingredients"):
- Factuality: The text makes things up or gets facts wrong (hallucinations).
- Bias: The text sounds too robotic or neutral when it should have a human voice, or it takes a weird, one-sided stance.
Style Quality (The "Bad Presentation"):
- Repetition & Templatedness: The AI keeps saying the same thing or uses the exact same sentence structure over and over, like a broken record.
- Verbosity: It uses 100 words to say what could be said in 10. It's "wordy" without being "wise."
- Coherence & Tone: The ideas don't flow logically, or the tone feels weird (like a robot trying to tell a joke).

2. The Human Test (Annotating the Slop)

To see if their checklist worked, they hired professional copy-editors to read 150 news articles and 100 Q&A passages. They asked the editors to highlight specific "sloppy" parts of the text.

The Surprise:
Even the experts didn't always agree on what was "slop."

The Subjectivity Problem: One editor might think a text is "slop" because it's too wordy, while another thinks it's fine.
The Connection: However, when the editors did agree a text was slop, it was usually because the text was missing relevance (didn't answer the prompt) or density (was too empty).
The Domain Difference:
- In News, slop was mostly about bad style and tone (too formal, too repetitive).
- In Q&A, slop was mostly about getting facts wrong or being structurally messy.

3. Can Machines Spot It? (The Automatic Check)

The researchers tried to see if current AI tools could automatically flag this slop without human help. They tested three things:

Standard Math Metrics: They used old-school math tools (like counting word length or checking for repeated words).
- Result: These tools were okay at spotting "wordy" text, but they failed to catch the subtle stuff like "is this relevant?" or "does this make sense?"
AI Reward Models: They used advanced AI models trained to grade writing quality.
- Result: These models could tell the difference between "good" and "bad" writing generally, but they couldn't specifically identify "slop" as defined by the humans. They missed the nuance.
AI Judges (LLMs as Judges): They asked powerful AI models (like GPT-5 and others) to read the text and say, "Is this slop?" and "Highlight the bad parts."
- Result: They failed miserably. The AI judges were terrible at this. They often missed the slop entirely or highlighted the wrong parts. They tended to focus only on "wordiness" and ignored everything else.

The Bottom Line

The paper concludes that while we have a good human-defined checklist for what "slop" looks like (it's mostly about being irrelevant, empty, or repetitive), we do not yet have a reliable robot that can automatically find it.

Currently, if you want to know if text is "slop," you still need a human to read it and use their judgment. The automatic tools are like a metal detector that only finds big rocks but misses the tiny, valuable gems (or the tiny, annoying pieces of trash) hidden in the sand.

What the paper does NOT claim:

It does not say we should ban AI writing.
It does not claim this method will fix AI in the future (it just says it's an open challenge).
It does not suggest using this for medical or legal decisions.
It strictly focuses on defining the problem and showing that current automated solutions aren't ready to solve it yet.

Technical Summary: Measuring AI "Slop" in Text

Problem Statement
The term "slop" has emerged to describe generic, low-quality content generated by Large Language Models (LLMs), characterized by non-factual claims, low information density, and structural quirks. Despite the ubiquity of the term in public discourse and its relevance to the growing use of AI in writing and information gathering tasks, there is no agreed-upon definition or standardized method for measuring "slop." Current AI-text detection methods focus on distinguishing human from machine origins, whereas "slop" is a quality metric applicable to any text source. Furthermore, existing text quality metrics (e.g., BLEU, ROUGE) and automatic evaluation models often fail to capture the specific stylistic and substantive deficits that constitute "slop," such as vagueness, lack of perspective, or structural templatedness.

Methodology
The authors employ a multi-stage approach to conceptualize, operationalize, and evaluate "slop":

Taxonomy Development: Through interviews with 19 experts in NLP, writing, journalism, linguistics, and philosophy, the authors derived a working definition and taxonomy of "slop." Using qualitative content analysis and deductive coding, they mapped expert definitions to measurable concepts, categorizing them into three themes:
- Information Utility: Density (substantive content relative to length) and Relevance (alignment with task/prompt).
- Information Quality: Factuality (accuracy/hallucinations) and Bias (presence/absence of necessary subjectivity).
- Style Quality: Structure (Repetition, Templatedness), Coherence, Tone, Fluency, Verbosity, and Word Complexity.
Data Collection and Annotation: The framework was validated using span-level annotations on two datasets:
- News Articles: 150 articles (human-written vs. AI-generated by Claude 3.5, GPT-4o, o1-pro) from Russell et al. (2025).
- Retrieval-Augmented QA: 100 passages from MS MARCO with responses generated by Llama-4, OLMo-2, Mistral-7B, GPT-4o, and Gemma-2.
- Professional copy-editors annotated spans for specific "slop" codes. Inter-annotator agreement was measured using Cohen's $\kappa$ , Gwet's AC1, and Krippendorff's $\alpha_{MASI}$ .
Analysis of Latent Dimensions: Logistic regression models were fitted to determine which annotated span-level codes predict binary "slop" judgments. The study analyzed correlations between these human judgments and automatic metrics.
Automatic Measurement Evaluation: The authors evaluated the efficacy of:
- Linear Models: Using standard automatic text metrics (e.g., compression ratios, subjectivity lexicons, readability scores) to predict "slop."
- Reward Models: Testing the Writing Quality Reward Model (WQRM) against the "slop" labels.
- LLM-as-a-Judge: Prompting GPT-5, Deepseek-V3, and o3-mini to identify binary "slop" labels and extract "slop" spans in zero-shot and few-shot settings.

Key Results

Subjectivity and Agreement: Binary "slop" judgments are somewhat subjective, with annotators showing poor to fair agreement on overall document labels ( $\kappa \approx -0.15$ to $0.29$). However, agreement improves when correcting for prevalence (AC1) and when focusing on specific codes. Factuality, Bias, and Structure codes achieved reliable annotation ($AC1 > 0.5$), while cognitively demanding constructs like Coherence, Density, and Relevance showed moderate-to-challenging agreement.
Predictive Validity: The number of annotated "slop" spans strongly correlates with overall "slop" judgments ( $\rho \approx 0.63$ ). Across domains, Relevance, Density, and Tone were the strongest predictors of a text being labeled "slop."
Domain Specificity: The importance of specific dimensions varies by task. For news articles, style quality (Coherence, Tone) and utility (Density, Relevance) are significant. For QA tasks, Factuality and Structure are the primary drivers of "slop" judgments.
Failure of Automatic Methods:
- Linear Models: While linear models using available automatic metrics achieved an AUPRC roughly double the prevalence baseline (0.52 for News, 0.55 for MS MARCO), they are insufficient for fully capturing the signal. Notably, 3 out of 5 significant "slop" features (Relevance, Factuality, Coherence) lack reliable automatic metrics.
- Reward Models: The WQRM aligns partially with "slop" labels but fails to fully capture the concept, showing low correlation with binary "slop" judgments (0.15–0.25).
- LLM Judges: Capable reasoning LLMs (GPT-5, o3-mini) fail to reliably identify "slop." They significantly under-predict the "slop" label compared to humans (0.03–0.08 vs. 0.34) and struggle with span extraction (Precision $\approx$ 0.08–0.14). Furthermore, LLMs tend to over-represent "Density" issues in their rationales while under-representing Coherence, Tone, and Bias.

Significance and Claims
The paper claims to provide the first systematic attempt at qualitatively characterizing "slop" in LLM-generated text through a validated taxonomy. The authors argue that "slop" is distinct from simple AI detection; it is a measure of quality that can apply to human writing but is particularly prevalent in AI outputs.

The study highlights that current automated evaluation practices are insufficient for measuring "slop." Standard metrics fail to capture whether text is genuinely useful or well-written relative to the specific deficits identified in the taxonomy. While the authors demonstrate that an increase in issues along the identified axes (Information Utility, Quality, and Style) increases the likelihood of a text being judged as "slop," they conclude that fully automated and scalable methods for measuring "slop" remain an open challenge. The proposed taxonomy serves as a framework for guiding future improvements in LLM-based reward models and quality assessment, particularly for subjective tasks where human preference judgments are critical.

1. Defining the "Slop" (The Taxonomy)

2. The Human Test (Annotating the Slop)

3. Can Machines Spot It? (The Automatic Check)

The Bottom Line

More like this