💬 NLP

CAVEWOMAN: How Large Language Models Behave Under Linguistic Input and Output Compression

The "Cavewoman" study reveals that while compressing a model's output can significantly reduce inference costs, compressing user input is counterproductive, as it forces models to generate longer responses that increase net costs and degrade accuracy.

Original authors: Morayo Danielle Adeyemi, Ryan A. Rossi, Franck Dernoncourt

Published 2026-06-24

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Morayo Danielle Adeyemi, Ryan A. Rossi, Franck Dernoncourt

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a very smart, but expensive, personal assistant to solve problems for you. You pay them based on how much they read (input) and how much they write (output). Usually, writing costs much more than reading.

The paper "CAVEWOMAN" investigates a popular idea: What if we force the assistant to speak like a caveman? (e.g., "Talk short. Drop grammar. Save token.") The goal is to save money by making the conversation shorter.

The researchers set up a clever experiment to see if this actually works. They tested two different ways to use "caveman speak" on eight different AI models across five types of puzzles.

Here is what they found, explained simply:

1. The Two Ways to Speak "Caveman"

The researchers tested two different channels, like two different ways to give instructions:

Channel A (The "Broken Prompt"): You give the AI a question that has been stripped of all its "glue words" (like "the," "is," "and," "to").
- Example: Instead of "What is the capital of France?" you type "Capital France?"
- The Result: This is a trap. It actually costs more money. Why? Because when the AI gets a broken, confusing question, it panics and writes a much longer, more detailed answer to try to figure it out. Since writing is expensive, the extra length costs more than the few words you saved on the question. It's a "lose-lose."
Channel B (The "Caveman Order"): You give the AI the full, normal question, but you tell it: "Answer me like a caveman. No grammar, just keywords."
- Example: You ask, "What is the capital of France?" but the AI replies, "Paris."
- The Result: This saves money. Because the AI is forced to be brief in its answer, it writes fewer words. Since writing is the expensive part, this actually cuts the bill significantly (sometimes by half or even two-thirds).

2. The "Ghost in the Machine" Problem

Here is the most surprising finding. When the AI is forced to answer like a caveman (Channel B), it often gets the right answer, but the words it uses are totally different from how it would normally explain itself.

The Analogy: Imagine a chef who usually writes a beautiful, 10-page recipe for a cake. You tell them, "Just give me the ingredients list." They hand you a list that says "Flour, Sugar, Eggs."
- The list is correct (you can bake the cake).
- But the list is not the same as the beautiful recipe they would have written if you hadn't forced them to be brief.
The Paper's Claim: About 52% of the time, the AI gets the answer right, but the "caveman" version of the answer is so different from its normal "full sentence" version that a computer can't tell they are saying the same thing.
Why it matters: If you only care about the final answer (like "Paris"), this doesn't matter. But if you need to read the AI's reasoning or keep a record of its thought process, the "caveman" version has lost the original flavor and logic, even if the result is correct.

3. Not All AIs Are the Same

The researchers found that some AIs are much better at handling these "caveman" rules than others.

Some models (like the open-source "Gemma-4") are very robust; they can still give good answers even when forced to be brief.
Other models (like the very smart "GPT-5.4") actually get much worse when forced to be brief, even though they are usually the best at normal tasks.
The Lesson: You can't just pick the "smartest" AI. You have to test which AI works best under the specific "caveman" rules you plan to use.

Summary

Don't break the question: If you strip words from the question you ask, the AI gets confused, writes more, and you pay more.
Do break the answer: If you tell the AI to give short answers, you save a lot of money.
Watch out for the "Ghost": Short answers might be correct, but they often look completely different from the AI's normal way of thinking. If you need to see the "why" behind the answer, the short version might be misleading.

The paper concludes that to save money and get good results, you should compress the output (the answer), not the input (the question), and you must check if the AI is still "thinking" correctly, not just if it got the right number.

Technical Summary: CAVEWOMAN – How Large Language Models Behave Under Linguistic Input and Output Compression

Problem Statement

Inference costs for Large Language Models (LLMs) scale with both input and output token counts, with output tokens typically priced 4–8× higher than input tokens. While compression methods exist to reduce either the prompt (input) or the response (output), they are traditionally evaluated in isolation using a single metric: task accuracy at a reduced token count.

This paper identifies two critical flaws in existing evaluation protocols:

Failure to measure realized cost: A shorter prompt does not necessarily reduce total inference cost if the model compensates by generating a longer response.
Binary accuracy collapse: Standard accuracy metrics collapse generations into binary outcomes, failing to distinguish between a compressed answer that aligns with the model's unconstrained reasoning and one that diverges from it while still being "correct."

Methodology: The CAVEWOMAN Protocol

The authors introduce CAVEWOMAN, a two-channel evaluation protocol designed to score every generation on three axes: task accuracy, realized per-item cost, and reference-text agreement against the model's own unconstrained generation.

Experimental Design

Models: Eight models were evaluated, including open-weight models (Qwen2.5-VL-7B, Qwen3.5-9B, DeepSeek-R1-Distill-Qwen-7B, Gemma-4-E4B) and API models (GPT-4o, GPT-5.4, Claude Haiku 4.5, Claude Sonnet 4.6).
Datasets: Five benchmarks covering four task types: GSM8K (math), BoolQ (boolean), ARC-Easy (science MCQ), CommonsenseQA (commonsense MCQ), and MMLU-STEM (STEM MCQ).
Compression Levels: Five levels of linguistic reduction ( $L_0$ $L_{0}$ to $L_4$ $L_{4}$ ) were applied using a deterministic Part-of-Speech (POS) filter:
- $L_0$ : Unconstrained baseline.
- $L_1$ : Telegraphic (removes closed-class words like determiners, prepositions, conjunctions).
- $L_2$ : Keyword-only (nouns, verbs, cardinal numbers).
- $L_3$ : Noun-phrase skeleton (nouns and cardinal numbers).
- $L_4$ : 15-token budget.
Two Conditions:
- Condition A (Input Compression): The user prompt is filtered via the POS filter before the model sees it.
- Condition B (Output Constraint): The prompt remains intact, but the system prompt instructs the model to generate a response in the corresponding reduced register.

Evaluation Metrics

Task Accuracy: Regex-extracted answers matched against ground truth.
Realized Cost: Calculated as $C = n_{in} \cdot p_{in} + n_{out} \cdot p_{out}$ , measuring the actual token cost incurred.
Reference-Text Agreement: Measured via bidirectional Natural Language Inference (NLI) entailment against the model's unconstrained ( $L_0$ ) generation on the same item. This is supplemented by 11 complementary semantic measures.
Dissociation Table: A 2x2 matrix crossing task correctness with reference-text agreement to identify cases where answers are correct but semantically divergent from the unconstrained baseline (Cell C2).

Key Findings

1. Cost Asymmetry Between Channels

Output Compression (Condition B): Successfully reduces realized cost on most API models (1.4–2.4× reduction per model, up to 3× in the best case) and all four open-weight models. This is because the constraint directly limits the expensive output tokens.
Input Compression (Condition A): Results in a "strict lose-lose." It raises net cost (approx. 1.15× on average, up to 1.8× on specific datasets, and 2.7× under deep compression) because models compensate for the stripped prompt by generating longer responses. Since output tokens are priced higher, the savings on input tokens are outweighed by the cost of expanded output.
Mechanism: Input compression triggers "compensatory output expansion," where the model attempts to reconstruct missing context, leading to higher total costs even as accuracy collapses at deeper reduction levels.

2. Decoupling of Accuracy and Reference-Text Agreement

Under output compression, a significant divergence emerges between task correctness and the model's internal reasoning trace:

Surface Text Divergence: On non-reasoning models, 51.9% of generations at the telegraphic level ( $L_1$ ) are correct but their surface text no longer entails the model's unconstrained ( $L_0$ ) generation.
Robustness: This divergence persists even when controlling for length (re-scoring truncated $L_0$ generations to match $L_1$ length) and holds across multiple semantic metrics and judges.
Interpretation: The model can arrive at the correct answer via a different reasoning path or surface formulation when constrained, meaning the "correctness" of the answer does not guarantee the preservation of the original reasoning trace.

3. Variable Robustness Across Models

Robustness to output compression (the ratio of output-constrained accuracy to input-constrained accuracy) varies widely across models (1.5× to 3.7× range).
This robustness is not predicted by parameter count or unconstrained baseline accuracy. For instance, the highest-accuracy API model (GPT-5.4) was among the least robust to output constraints, while a 7B open-weight model (Gemma-4-E4B) showed the highest robustness.

Significance and Claims

The paper argues that single-axis evaluation of compression is underdetermined. The authors claim:

Cost cannot be inferred from prompt reduction: Realized cost depends on the interaction between input reduction and output expansion.
Accuracy is conditional: Observed accuracy depends on extraction reliability and the specific semantic axis used for evaluation.
Deployment implications: For deployments consuming only the final answer, output compression is a viable cost-saving strategy. However, for deployments requiring audit trails or reasoning transparency (where the generation text itself is consumed), output compression introduces a significant risk of semantic divergence from the model's unconstrained reasoning.
Evaluation Protocol: Candidates for constrained deployment should be ranked based on their performance at the specific constraint level they will face, rather than their unconstrained baseline performance.

The study concludes that compressing LLM inference is a two-channel problem where input and output compression yield fundamentally different economic and semantic outcomes, necessitating multi-metric evaluation protocols like CAVEWOMAN.