💬 NLP

Every Time I Hire a Linguist, Inference Costs Go Down: On Linguistic Rules as Effective Prompt Compressors

This paper demonstrates that deterministic linguistic rules, discovered through evolutionary search, can serve as effective, low-cost prompt compressors that reduce LLM inference expenses without requiring model forward passes during compression, achieving performance comparable to advanced LM-based methods.

Original authors: Jianfei Ma, Zhaoxin Feng, Emmanuele Chersoni, Si Chen

Published 2026-07-29

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Jianfei Ma, Zhaoxin Feng, Emmanuele Chersoni, Si Chen

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a super-smart robot how to read a massive library of books to answer a single question. The robot is brilliant, but it has a problem: it gets overwhelmed by too much text. If you feed it a whole novel, it slows down, costs a fortune to run, and sometimes misses the point because the important details get lost in the noise. This is the world of "Large Language Models" (LLMs), the AI brains behind many modern tools. To fix this, scientists have been trying to "compress" the text—shaving off the fluff to keep only the meaty facts.

Until now, the standard way to do this has been like hiring a very expensive, slow editor who reads every single word and calculates a "score" for how important it is. This editor is actually another AI model running in the background. It's accurate, but it's heavy, slow, and requires powerful, expensive computer chips (GPUs) to work. It's like using a gold-plated scalpel to trim a hedge.

But what if you didn't need a gold-plated scalpel? What if you could use a simple, old-fashioned pair of garden shears based on a few clear rules? This is the big question the paper asks: Can we compress text using simple, deterministic linguistic rules—like "keep the nouns, drop the 'the's and 'and's"—without needing a second AI to score every word?

The researchers, a team from The Hong Kong Polytechnic University, decided to find out. They didn't just guess; they set up a digital "evolutionary search." Imagine a virtual jungle where they planted 42 different "seeds" of compression rules. These seeds were based on centuries of human linguistic knowledge: some looked at word types (like keeping "engineer" but dropping "the"), some looked at sentence structure (keeping the main action but cutting the extra descriptions), and some looked at the flow of the story.

Then, they let an AI act as a "mutator." It took these rule-seeds, mixed them up, tweaked them, and created new combinations. It tested these new "compressors" by seeing if a robot could still answer questions correctly after the text was shrunk. The fittest compressors—the ones that kept the most important info while cutting the most words—survived and bred the next generation.

The result? They discovered that a compressor built entirely from these linguistic rules could perform just as well as the heavy, expensive AI-based methods for many types of long documents. The best part? Once the compressor was "evolved," it didn't need any AI to run. It could run on a standard computer processor (CPU) in a flash, costing almost nothing. It was like swapping that slow, gold-plated editor for a lightning-fast, rule-based machine that never gets tired.

However, the paper also found a limit. These rule-based shears work brilliantly when you need to cut the text down by a little bit or a moderate amount (like shrinking a 100-page document to 25 pages). But if you try to be too aggressive and cut the text down to a tiny fraction (like 10 pages), the rules start to struggle. They can't be as clever as the expensive AI editor when the budget is extremely tight.

In short, the paper suggests that for many real-world tasks, we don't need to overcomplicate things with expensive AI scoring. Sometimes, a well-designed set of linguistic rules, honed by evolution, is the perfect, lightweight tool to make AI faster, cheaper, and just as smart.

Technical Summary: "Every Time I Hire a Linguist, Inference Costs Go Down: On Linguistic Rules as Effective Prompt Compressors"

1. Problem Statement

Large Language Models (LLMs) face increasing inference costs and latency when processing long-context applications such as Retrieval-Augmented Generation (RAG), multi-agent communication, and long-term dialogue. Current "hard prompt compression" methods, which prune tokens while maintaining readability, predominantly rely on Language Model (LM) scoring (e.g., perplexity, trained classifiers, or attention-based signals) to determine token importance. This approach presents two critical limitations:

Computational Overhead: LM-based scoring requires forward passes during compression, negating some of the inference cost savings intended by compression.
Granularity Risks: Token-level selection based on opaque model scores risks severing syntactic dependencies and semantic coherence.

The authors question whether fine-grained, per-token model scoring is necessary for effective compression, proposing instead that deterministic linguistic rules—long studied in linguistics for identifying informative content—could serve as effective compressors without requiring an LM forward pass at deployment.

2. Methodology

The paper introduces a framework for discovering prompt compressors using offline evolutionary search over explicit linguistic rules.

Seed Construction: The initial population consists of 42 "seeds" encoding deterministic rules across four linguistic levels:
- Lexical: Content vs. function word distinctions, named entity preservation, and word frequency (Zipf's law).
- Syntactic: Dependency structure analysis to extract core argument triples (e.g., subject-verb-object) and prune dispensable subtrees (e.g., relative clauses, adjectival modifiers).
- Semantic: Preservation of negation markers, modal verbs, quantifiers, and comparative constructions.
- Discourse: Sentence position, discourse connectives, entity chains, and graph-based sentence ranking.
- Composite: Pipelines chaining rules from multiple levels.
Evolutionary Search: The authors adapt AlphaEvolve to navigate the combinatorial space of these rules.
- Process: An LLM acts as a "mutator," editing Python scripts that implement the compression logic. The search is conducted across six independent "islands" with periodic migration to maintain diversity.
- Evaluation: A dual-path protocol evaluates candidates:
  1. Direct Path: The receiver LLM answers questions directly from the compressed text.
  2. Reconstruction Path: A reconstruction LLM expands the compressed text back into prose, which is then fed to the receiver LLM.
- Fitness Function: $f = \min(D, R)$ , where $D$ is Direct accuracy and $R$ is Reconstruction accuracy. This conservative metric ensures the compressor retains information sufficient for both direct inference and faithful reconstruction.
- Constraints: The search uses a cascaded filtering system (syntax check, coarse QA, strict QA) to manage computational cost. Crucially, the final evolved compressors rely exclusively on standard NLP pipelines (e.g., spaCy) and require no LM forward pass during deployment.

3. Key Results

The evolved compressors were evaluated on four datasets (RACE, Qasper, Multi-doc QA, LongMemEval) across compression ratios of 1.5×, 2.5×, 4×, and 6×, compared against eight state-of-the-art baselines (e.g., LLMLingua, PartPrompt, FrugalPrompt).

Performance: In medium-to-long document settings, the evolved linguistic compressors achieve performance similar to recent advanced prompt-compression strategies.
- They rank first or second on Qasper and Multi-doc QA at light-to-moderate compression (1.5×–2.5×).
- Performance degrades as compression becomes aggressive (6×), where the granularity gap between fixed linguistic rules and continuous LM scoring becomes more pronounced.
- On RACE (short passages), the method trails stronger baselines, likely due to a training bias toward longer-text patterns.
Stability: The method exhibits low cross-model variability (measured by Coefficient of Variation) on RACE and Multi-doc QA, indicating robustness across different receiver LLMs (Qwen, LLaMA, Gemma).
Efficiency: The deployed compressors run entirely on CPU with negligible latency (e.g., ~8–33 ms per input) and low memory footprint, contrasting sharply with baselines that require GPU-based LM inference.
Controllability: Unlike LM-based baselines that often drift from target compression ratios, the rule-based approach enforces exact word budgets, achieving near-zero deviation from target ratios.
Transferability: The compressors generalize to Out-of-Distribution (OOD) tasks (OOD QA, summarization, classification) without re-evolution, performing competitively with the strongest baselines.

4. Analytical Insights

Evolutionary analysis reveals distinct patterns in how effective compression is achieved:

Cross-Level Fusion: The best compressors consistently fuse signals from lexical, syntactic, semantic, and discourse levels rather than relying on a single level.
Granularity Shift: As the compression ratio increases, the dominant unit of compression shifts from token-level pruning (at 1.5×) to sentence-level extraction (at 6×).
Rule Evolution: The search eliminates certain theoretically motivated rules (e.g., query-aware matching, synonym substitution) that prove redundant in the final combination, while retaining core signals like named entities, negation, and discourse connectives.

5. Significance and Claims

The paper claims three primary contributions:

Framework: It introduces one of the first prompt compression frameworks centered on explicit linguistic rules evolved via LLM-guided search, moving away from opaque, LM-dependent scoring.
Empirical Findings: It demonstrates that rule-based compressors can achieve performance comparable to advanced learned strategies in many long-document settings, offering computationally efficient deployment (CPU-only, no forward pass) and high stability across models.
Analytical Insights: It provides evidence that effective compression requires fusing multiple linguistic levels and that the optimal granularity shifts from tokens to sentences as compression budgets tighten.

The authors maintain a modest stance, acknowledging limitations such as dependence on English-specific parsers (spaCy), the lack of exhaustive search over all possible linguistic rules, and the asymmetry in optimization budgets compared to baselines trained on general data. They conclude that while linguistic rules alone may not universally dominate learned methods under extreme compression, they offer a highly interpretable and lightweight alternative for long-context applications.