💬 NLP

Researchers waste 80% of LLM annotation costs by classifying one text at a time

This study demonstrates that researchers can reduce LLM annotation costs by over 80% through batching and variable stacking without significantly compromising coding accuracy, as long as batch sizes and prompt complexity remain within specific safe operating ranges where measurement error stays below typical inter-coder disagreement.

Original authors: Christian Pipal, Eva-Maria Vogel, Morgan Wack, Frank Esser

Published 2026-04-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Christian Pipal, Eva-Maria Vogel, Morgan Wack, Frank Esser

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher grading a stack of 100,000 student essays. You need to check four things for each essay: Is it on topic? Does it have a clear argument? Is the tone positive or negative? What is the main subject?

The Old Way (The "One-by-One" Method):
Currently, most researchers using AI to do this work act like a very strict, slow teacher. They pick up one essay, ask the AI, "Is this on topic?" The AI answers. Then, they put that essay down, pick up the same essay again, and ask, "Is the tone positive?" The AI answers again. They repeat this for every single variable for every single essay.

The paper argues that this is like paying a taxi driver to drive you one block, drop you off, and then drive you back to the exact same spot just to pick up your bag before driving you one block further. You are paying for the "pickup and drop-off" (the API call) over and over again. If you have 100,000 essays and 4 questions, you are making 400,000 trips. This is incredibly expensive and wasteful.

The New Way (The "Batching and Stacking" Method):
The researchers discovered you can be much smarter. Instead of asking one question about one essay, you can:

Batch: Hand the AI a whole stack of 25 essays at once.
Stack: Ask the AI to check all 4 questions for all 25 essays in a single go.

The Analogy: The Super-Scanner
Think of the AI not as a person reading a book, but as a super-fast barcode scanner at a grocery store.

The Old Way: You walk up to the counter, scan one apple. Then you walk up again and scan the same apple to check its weight. Then you walk up again to check its color. You do this for every apple in the cart.
The New Way: You put the whole cart on the conveyor belt. You tell the scanner, "Check the price, weight, and color of all these 25 items right now." The scanner zips through them in a flash.

What Did They Find?
The researchers were worried: "If we ask the AI to do too much at once, will it get confused and make mistakes?" They tested this with 8 different "brains" (AI models) and nearly 4,000 real tweets.

Here are the results, translated into plain English:

The "Sweet Spot" (Batch Size 25–100):
You can safely feed the AI a batch of 25 to 100 items at a time. The AI doesn't get confused. It stays just as accurate as if you asked it one by one.
- The Metaphor: It's like a chef who can chop 25 onions at once just as perfectly as chopping one. They don't get tired or sloppy until you give them a mountain of 1,000 onions.
The "Multi-Task" Magic (Stacking Variables):
You can ask the AI to check up to 10 different things (variables) in a single prompt without losing quality.
- The Metaphor: It's like a security guard who can check your ID, your bag, and your shoes all at the same time, rather than making you stand in three different lines.
The Cost Savings:
By using this method, researchers can cut their costs by over 80%.
- The Metaphor: If the old way cost you $337 to analyze a huge dataset, the new way costs you less than $50. That's like buying a luxury dinner and getting a free dessert, or buying a single movie ticket and getting a whole season of a show.
The Only Catch:
Some specific AI models (the "reasoning" ones from OpenAI) got confused if the pile got too big (over 250 items). But for most standard models, the "sweet spot" is huge. Also, the AI gets slightly more tired if the questions are very complex, but not because the prompt is long—it's just because the job is hard.

The Bottom Line:
For a long time, researchers were being overly cautious, treating the AI like a fragile human who needs a break after every single task. This paper proves the AI is actually a high-speed machine that loves to work in bulk.

By simply changing how we ask questions (grouping them together), we can save massive amounts of money and time without sacrificing the quality of the answers. The "error" introduced by doing this is so small that it's actually smaller than the disagreements you'd get if you hired two different human experts to do the same job.

In short: Stop asking the AI to do one thing at a time. Give it a whole pile of work, and let it crush it efficiently.

1. Problem Statement

Large Language Models (LLMs) have become standard tools for automated content analysis in the social sciences, capable of classifying millions of texts. However, current implementation practices are highly inefficient. Researchers typically adopt a conservative approach: one text per variable per prompt.

The Cost: Classifying 100,000 texts on four variables using this baseline requires 400,000 API calls.
The Inefficiency: This method repeats instructions (system prompts and few-shot examples) for every single text, leading to massive token waste.
The Knowledge Gap: While computer science literature suggests batching is viable for small NLP tasks, it is unknown whether Item Batching (grouping multiple texts in one prompt) and Variable Stacking (requesting multiple coding dimensions in one prompt) degrade classification accuracy in social science contexts. If these methods work, they could reduce costs by over 80% without compromising data quality.

2. Methodology

The authors conducted a systematic experiment involving 3,962 expert-coded tweets across four classification tasks.

Data Source: Replication data from Gilardi et al. (2023), containing tweets about content moderation.
- Variables: Relevance (binary), Problem/Solution (3-class), Stance on Section 230 (3-class), and Topic (6-class).
- Ground Truth: Human-coded labels with inter-coder reliability ( $\kappa$ ) ranging from 0.63 to 0.88.
Models Tested: Eight production LLMs from four providers:
- Anthropic: Claude Haiku 4.5.
- OpenAI: GPT-5-mini, GPT-5-nano (Reasoning models, locked at Temp=1).
- Google: Gemini 3 Flash, 3.1 Flash-Lite, 2.5 Flash-Lite.
- Alibaba: Qwen 3.5 Plus, Qwen 3.5 Flash.
Experimental Design:
- Study 1 (Item Batching): Varied batch sizes ( $b$ ) from 1 to 1,000 items per prompt.
- Study 2 (Variable Stacking): Varied the number of simultaneous coding dimensions ( $k$ ) from 1 (baseline) to 4, 10, and 25.
- Control: A prompt with the same token length as the $k=25$ condition but containing only 4 variables (plus filler text) to isolate the effect of task complexity vs. prompt length.
Total Scale: 376 experimental conditions resulting in 963,151 individual classifications and ~96,000 API calls.

3. Key Results

A. Item Batching (Grouping Texts)

Stability: Six of the eight models maintained accuracy within 2 percentage points (pp) of the single-item baseline ( $b=1$ ) even at a batch size of 100.
Cost Savings: Moving to $b=100$ resulted in ~84% cost savings (token reduction) with negligible accuracy loss.
Model Sensitivity:
- Robust: Claude Haiku 4.5 showed the highest stability (max drop of 1.1 pp).
- Sensitive: Google and Alibaba models showed moderate degradation (3–4 pp) at $b=1000$ .
- Failure Cases: Two OpenAI reasoning models (GPT-5-nano, GPT-5-mini) collapsed at $b \ge 250$ , losing 27–36 pp in accuracy due to parse failures and inability to handle large batches at their locked temperature settings.
Variable Specifics:
- Relevance was the most sensitive to batching.
- Topic classification actually improved at larger batch sizes for some models, suggesting co-presented tweets provide implicit distributional information that helps calibrate category boundaries.

B. Variable Stacking (Grouping Dimensions)

Viability: Stacking up to 10 dimensions produced results comparable to single-variable coding (within 2.2 pp of baseline).
Limits: Stacking 25 dimensions caused a drop of 1.6–6.1 pp for most models (GPT-5-nano dropped 15.5 pp).
Interaction: The effects of batching and stacking were additive, not multiplicative. Researchers can optimize both dimensions simultaneously without a compounding penalty.

C. Prompt Length vs. Task Complexity

The control experiment proved that accuracy degradation in stacking is driven by task complexity (managing multiple classification schemes simultaneously), not by the sheer length of the prompt (token count). A prompt with 25 tokens of filler text performed as well as a short prompt with 4 variables.

4. Key Contributions

Quantified Efficiency: Demonstrated that batching 25 items and stacking 4 variables reduces API calls from 400,000 to 4,000 for a 100k-text project, cutting costs by >80%.
Defined Safe Operating Ranges: Established that for most current LLMs, a batch size of 25–100 and variable stacking of up to 10 dimensions are safe, with accuracy loss smaller than typical human inter-coder disagreement (6–21%).
Disentangled Factors: Proved that prompt length is not the primary driver of error; rather, the cognitive load of managing multiple distinct classification tasks is the limiting factor.
Model Differentiation: Highlighted that model choice matters more than batch size. The performance gap between the best and worst models at a fixed batch size exceeds the performance drop caused by increasing batch size.

5. Significance and Implications

Feasibility of Large-Scale Studies: The findings make large-scale LLM-based content analysis financially feasible for researchers with limited budgets. For example, coding 100,000 tweets could drop from ~$337 to under $50 (or <$10 with budget models).
Methodological Standard: The paper argues that batch size and stacking configuration should be reported alongside model names and prompts, similar to how hyperparameters are reported in machine learning.
Error Tolerance: The measurement error introduced by optimal batching/stacking is statistically smaller than the inherent noise in human-coded ground truth data, meaning these efficiency gains do not compromise scientific validity.
Caveats: The results are based on short English tweets. Generalization to long documents, other languages, or highly complex reasoning tasks requires further investigation.

Conclusion: Researchers are currently wasting significant resources by processing texts one-by-one. By adopting batch sizes of 25–100 and stacking up to 10 variables, the social science community can achieve massive cost reductions with negligible impact on data quality, provided they select robust models and avoid the specific failure modes of certain reasoning architectures.