💻 computer science

Tool-Schema Compression Enables Agentic RAG Under Constrained Context Budgets

This paper demonstrates that tool-schema compression is a critical infrastructure for Agentic RAG systems, enabling functional performance under constrained context budgets by preventing tool definitions from overflowing the context window and significantly improving exact-match scores compared to uncompressed schemas.

Original authors: Furkan Sakizli

Published 2026-05-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Furkan Sakizli

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a super-smart assistant (an AI) to solve a complex problem. To do this job, you give the assistant two things:

A massive instruction manual (the "Tool Schemas") that lists hundreds of tools the assistant can use, like "search the database," "calculate a budget," or "find a document."
A stack of reference papers (the "Retrieval Chunks") containing the specific facts needed to answer the question.

The problem is that the assistant has a limited desk space (the "Context Window"). It can only hold a certain amount of paper at once.

The Problem: The Desk is Too Small

In this paper, the researcher found a critical bottleneck. When the assistant is given a standard, verbose instruction manual (written in a format called "JSON Schema"), the manual itself takes up so much desk space that there is no room left for the reference papers.

The Analogy: Imagine trying to fit a 100-page instruction manual and a 50-page research report onto a desk that only fits 80 pages total. If you put the manual down first, the research report has nowhere to go. The assistant ends up staring at the manual, unable to see the facts it needs to solve the problem.
The Result: When the desk is small (8,000 tokens), the assistant fails almost completely. It gets a near-zero score because it literally cannot see the evidence.

The Solution: Compressing the Manual

The researcher tested a new method called TSCG (Tool-Schema Compression). Think of this as taking that 100-page instruction manual and rewriting it in a highly efficient, shorthand format that keeps all the important rules but cuts out the fluff.

The Analogy: Instead of a thick, heavy manual, you now have a slim, pocket-sized guide. It takes up half the space.
The Result: Suddenly, the desk has room! You can fit the compressed manual and the reference papers. The assistant can now read the facts and solve the problem.

The Key Findings (The "Binary Enablement")

The paper calls this a "Binary Enablement" effect. It's not a small improvement; it's an on/off switch.

Without Compression: The system is broken. The assistant is blind to the facts. (Score: ~2.6%)
With Compression: The system works. The assistant can see the facts and answer correctly. (Score jumps to ~23% or higher).

It's like the difference between a car with no gas (it won't move at all) and a car with just enough gas to drive to the store. You don't need a Ferrari engine; you just need enough fuel to get the car moving.

What Happens When the Desk is Huge?

The researcher also tested this on a giant desk (32,000 tokens) where there was plenty of room for both the manual and the papers, even without compressing the manual.

The Finding: On the giant desk, compressing the manual didn't really change the score. The assistant did just as well with the big manual as with the small one.
The Lesson: Compression isn't about making the AI "smarter." It's purely about making space. If you already have enough space, you don't need to compress. But if your space is tight, compression is the only way to make the system work.

The "Too Much Information" Trap

Interestingly, the paper found that giving the assistant too many reference papers (when the desk is medium-sized) can sometimes confuse smaller, less powerful AI models.

The Analogy: If you give a student a single, clear textbook page, they might get the answer. If you give them that same page plus 20 other irrelevant pages, they might get overwhelmed and miss the point.
The Finding: Compressing the manual sometimes frees up space for more reference papers. For small AI models, having too many extra papers actually made them perform slightly worse because they got distracted.

The Future: Scaling Up

Finally, the researcher tested what happens if you have hundreds of tools (like 800 different tools).

The Finding: With standard manuals, the desk overflows (breaks) at about 494 tools. With the compressed manuals, the system keeps working smoothly even past 800 tools.
The Takeaway: As companies try to give their AI assistants more and more tools, they will eventually hit a wall where the instructions take up all the memory. Compressing those instructions is the only way to keep the system running.

Summary

This paper proves that for AI assistants working with limited memory, compressing the tool instructions is not just a nice-to-have optimization; it is a necessity. Without it, the system crashes because there's no room for the facts. With it, the system springs to life. It's the difference between a locked door and an open one.

Technical Summary: Tool-Schema Compression Enables Agentic RAG Under Constrained Context Budgets

Problem Statement

Agentic Retrieval-Augmented Generation (RAG) systems face a critical resource conflict: the context window required for retrieval-augmented generation is consumed by the tool definitions necessary for the agent to function. In production environments, agents often require 20–100+ tool definitions, typically represented as verbose JSON Schema objects. These schemas consume 300–500 tokens per tool. Consequently, at modest context budgets (e.g., 8K tokens) with as few as 28 tools, the schema block alone (~11,000 tokens) exceeds the context window entirely. This "context overflow" leaves zero budget for retrieval chunks, system prompts, or conversation history, rendering the RAG system non-functional.

Existing research on prompt compression primarily targets natural language, failing to address the structural invariants (syntax, parameter types, enums) required for machine-readable tool definitions. Furthermore, prior work has not systematically evaluated the trade-off between schema size and retrieval performance in agentic workflows.

Methodology

The authors present the first controlled study of tool-schema compression in agentic RAG, utilizing a rigorous experimental design:

Benchmark: A purpose-built "NovaTech-28" benchmark featuring 28 tool definitions (database, document retrieval, computation, communication) and 100 questions across five categories (single-hop, multi-hop, tool-requiring, etc.). The retrieval corpus consists of 40 chunks (350 tokens each).
Compression Technique: The study employs TSCG (Tool-Schema Compression Generator), a deterministic, rule-based compression method that preserves JSON Schema syntax while achieving 44–50% token savings.
Experimental Scope:
- Models: 14 models ranging from 1.5B to 32B parameters (local) plus one frontier API model (Claude Sonnet 4).
- Context Budgets: Three levels: 8K, 16K, and 32K tokens.
- Scale: 6,566 controlled API calls.
- Design: Paired comparisons (same questions, same models) varying only the schema format (JSON vs. TSCG) and resulting context budget allocation.
Metrics: Primary metrics include Exact Match (EM) and Token F1. Secondary metrics include tool selection accuracy and context overflow detection. Statistical significance is assessed via Wilcoxon signed-rank tests and bootstrap confidence intervals.
Validation: External validation on HotpotQA (50 multi-hop questions) and frontier scaling tests up to 800 tools using a 200K context window.

Key Results

1. Binary Enablement at Constrained Budgets (8K)

At an 8K context budget with 28 tools, uncompressed JSON schemas cause 100% context overflow, resulting in near-zero Exact Match (EM) scores (average 2.6%).

Effect of Compression: Applying TSCG compression reduces schema tokens by ~50%, freeing enough space to fit at least one retrieval chunk.
Performance Lift: This restoration of functionality yields a massive average EM increase of +20.5 percentage points (pp) across all eight models tested at 8K. Models exhibiting "full enablement" saw an average lift of +24.7 pp.
Mechanism: The gain is attributed to converting the system from a state of total failure ( $k=0$ chunks) to a functional state ( $k \ge 1$ chunk), capturing the steepest portion of the context utilization curve.

2. Budget-Driven Effects vs. Format Intrinsic Effects

16K Budget: Both formats fit without overflow. TSCG allows for ~26 chunks compared to JSON's ~9 chunks. Despite the 3x increase in retrieval context, the accuracy difference is negligible (median $\Delta$ EM = -1 pp). This confirms the 8K gains are purely due to budget reallocation, not an intrinsic superiority of the compressed format.
32K Budget: Both formats comfortably fit all 40 chunks. Four of five models show $\Delta$ EM $\le$ 1 pp, confirming that when budget constraints are removed, the compression format has no significant impact on accuracy.

3. External Validation and Scaling

HotpotQA: On an independent dataset, TSCG enabled a +48 pp EM gain at 8K context, where JSON schemas failed completely.
Frontier Scaling: Tests with Claude Sonnet 4 at 200K context revealed that JSON schemas overflow at approximately 494 tools, causing total system failure. TSCG schemas remained operational beyond 800 tools, extending the operational range by 63%.

4. Failure Modes and Analysis

Distractor Dilution: In small models ( $\le$ 8B parameters) at 16K, providing more chunks via TSCG sometimes reduced accuracy. The authors attribute this to "distractor dilution," where limited attention capacity is overwhelmed by irrelevant context, echoing the "lost in the middle" phenomenon.
Context Utilization Model: The authors fit a descriptive model $C(k) = C_{max}(1 - e^{-\lambda k}) + C_0$ , showing that the marginal gain from the first retrieval chunk is the largest, with diminishing returns thereafter. This explains why recovering just one chunk at 8K yields disproportionate accuracy gains.

Significance and Claims

The paper reframes tool-schema compression from an optional cost optimization to a necessary infrastructure layer for agentic RAG in constrained-context deployments.

Categorical Enablement: The primary contribution is the discovery of a "binary enablement" effect. Without compression, agentic RAG at 8K is fundamentally broken (near-zero performance); with compression, it becomes functional. This is a categorical difference between a system that cannot operate and one that can be iteratively improved.
Lossless Fidelity: The study demonstrates that TSCG compression is lossless in practice; when context budgets are sufficient (32K), compressed and uncompressed schemas yield identical accuracy, proving the compression does not introduce accuracy artifacts.
Scalability Necessity: As tool counts grow in production (e.g., MCP deployments), schema compression becomes a practical necessity to prevent context overflow, extending the viable tool count by over 60% even in large context windows.

The authors conclude that for local model deployments and constrained environments, tool-schema compression is not merely a performance tweak but a prerequisite for enabling retrieval-augmented reasoning. All code, data, and checkpoints are publicly available.