💬 NLP

Structured Context Engineering for File-Native Agentic Systems: Evaluating Schema Accuracy, Format Effectiveness, and Multi-File Navigation at Scale

This paper presents a large-scale empirical study demonstrating that for file-native LLM agents, model capability is the dominant factor influencing performance, while the effectiveness of context engineering strategies like file-based retrieval and data formatting is highly dependent on the specific model tier rather than following universal best practices.

Original authors: Damon McMillan

Published 2026-02-13

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Damon McMillan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are hiring a brilliant but very literal super-intelligent assistant (an AI agent) to manage a massive library of information for you. Your goal is to ask this assistant to find specific books, check their contents, and write a summary based on what it finds.

This paper is a giant experiment to figure out the best way to organize that library so the assistant does its job perfectly. The researchers tested 11 different "assistants" (AI models) with 4 different ways of organizing the books (formats) and libraries ranging from a small bookshelf to a warehouse with 10,000 shelves.

Here is the breakdown of their findings, translated into everyday language:

1. The Two Ways to Hand Over the Library

The researchers tested two main strategies for how the assistant gets information:

The "Dump the Whole Library" Method (Prompt Engineering): You give the assistant the entire library's catalog in one giant stack of paper right at the start. It has to read through everything to find what it needs.
The "Librarian with a Flashlight" Method (File-Native Agents): You give the assistant a flashlight and a map. The library is huge, so the assistant uses a tool (like grep) to search for specific keywords and only pulls out the specific pages it needs, one by one.

The Big Surprise:

For the "Super-Genius" Assistants (Frontier Models like Claude, GPT-5): The "Flashlight" method worked better. These smart assistants are good at searching and picking out exactly what they need, so they didn't get overwhelmed by a giant stack of paper.
For the "Smart but New" Assistants (Open Source Models): The "Flashlight" method actually hurt them. These assistants got confused by the searching process. They performed better when you just gave them the whole stack of paper to read at once.
- Analogy: It's like giving a chess grandmaster a map to find a specific move (they love it), but giving a beginner a map makes them panic and lose the game.

2. Does the "Font" or "Paper Type" Matter? (The Format)

The researchers tried organizing the library in four different ways:

YAML: Like a neat, indented list.
Markdown: Like a blog post with headers and bold text.
JSON: Like a strict, computer-readable code block.
TOON: A super-compact, shorthand code designed to save space.

The Finding:
Overall, it didn't matter much which format you used. The assistants got the right answer about 75% of the time regardless of the format.

However, some specific assistants had strong preferences. One might hate the "shorthand" format (TOON) and prefer the "blog post" style (Markdown), while another might prefer the opposite.
Key Takeaway: Don't obsess over the format to boost accuracy. Instead, pick the format that is easiest for humans to maintain or that saves the most money on "paper" (tokens).

3. The "Brain Power" Gap is Huge

The single biggest factor in whether the assistant succeeded wasn't the library organization or the format—it was how smart the assistant was to begin with.

The "Super-Genius" models were about 21% more accurate than the "Open Source" models, no matter what you did.
Analogy: If you give a PhD student and a high schooler the same difficult math test, the PhD student will win every time, regardless of whether the test is printed in Comic Sans or Times New Roman. Invest in a smarter model first; worry about the file format later.

4. Scaling Up: The 10,000-Table Problem

What happens when the library grows from 10 books to 10,000?

The Problem: If you keep the whole library in one giant file, the assistant gets lost.
The Solution: The researchers split the library into sections (like "Fiction," "History," "Science").
The Result: Even with 10,000 tables, if the library is well-partitioned, the "Flashlight" method still works perfectly. The assistant just looks in the "History" section when asked about history, ignoring the rest.

5. The "Grep Tax" (The Hidden Cost of Being Too Compact)

This is a fascinating discovery about efficiency.

The Trap: You might think a super-compact format (like TOON) saves money because the files are smaller.
The Reality: Sometimes, being too compact makes the assistant work harder and spend more money.
Why? When the assistant uses its "flashlight" (search tool) to find a line in a compact file, that single line might be packed with so much information that it returns a huge chunk of text. Or, because the format is so new and weird, the assistant has to try searching in different ways (like trying different keywords) before it finds the right one.
Analogy: Imagine a dictionary where every definition is written in a tiny, dense font. You save paper, but your eyes get tired, and you have to squint and re-read the line three times to understand it. That "re-reading" costs you time and energy.

Summary: What Should You Do?

If you are building an AI system to manage data, here is the cheat sheet from the paper:

Pick Your Model First: If you have a budget for a top-tier "Super-Genius" model, use the File-Native (Flashlight) approach. If you are using a cheaper, open-source model, stick to Prompt Engineering (The Big Stack).
Don't Stress the Format: Use YAML if you want to save money on tokens and keep things tidy. Use Markdown if you want humans to be able to read it easily. Don't try to invent a new "super-compact" format unless you know exactly how your specific AI handles it.
Organize by Topic: If your data is huge, split it into folders (domains). Don't dump everything into one giant file.
Watch Out for "Shorthand": If you use a very compact, custom format, your AI might get confused and waste money searching for the right pattern. Stick to standard formats unless you are an expert.

The Bottom Line: There is no "one size fits all" magic trick. You have to match your library organization strategy to the intelligence level of the assistant you are hiring.

1. Problem Statement

As Large Language Model (LLM) agents increasingly interact with external systems via programmatic interfaces, practitioners lack empirical guidance on how to structure the context these agents consume. Specifically, there is a gap in understanding:

Whether file-native architectures (where agents retrieve schema information via tools like grep and read) outperform traditional prompt engineering (embedding full schemas in the system prompt).
How different schema representation formats (YAML, Markdown, JSON, TOON) affect accuracy and efficiency.
How model capability tiers (Frontier vs. Open Source) influence the effectiveness of these architectural choices.
How these systems scale to massive schemas (up to 10,000 tables) and whether "compact" formats actually reduce runtime costs.

2. Methodology

The study is a systematic, large-scale empirical evaluation involving 9,649 experiments across 11 models, 4 formats, 2 architectures, and schemas ranging from 10 to 10,000 tables.

Task: Text-to-SQL generation using the TPC-DS benchmark as a proxy for programmatic agent operations.
Architectures Compared:
- File Agent: Agents use native file tools (grep, read) to retrieve relevant schema sections on demand.
- Prompt Baseline: The complete schema (~6,000 tokens) is embedded directly in the system prompt with no tool access.
Formats Tested:
- YAML: Hierarchical, grep-friendly.
- Markdown: Documentation-style, natural language.
- JSON: Machine-parseable, verbose.
- TOON (Token-Oriented Object Notation): A custom compact format (~25% smaller than YAML) using abbreviated keywords.
Models: 11 models across three tiers:
- Frontier: Claude Opus, GPT-5, Gemini Pro.
- Frontier Lab: Claude Haiku, GPT-5 Mini, Gemini Flash.
- Open Source: DeepSeek, Kimi, Llama variants, Qwen.
Evaluation Metrics:
- Accuracy: Jaccard similarity between agent-generated and ground-truth result sets (Threshold $\ge$ 0.9).
- Efficiency: Token consumption, tool call counts, and runtime cost.
- Scale: Navigation accuracy across complexity tiers (L1-L5) and schema sizes (S0-S9, up to 10k tables).

3. Key Contributions

Systematic Scale: The largest study of its kind, covering 9,649 experiments across diverse models and schema sizes.
Architecture Validation: Demonstrated that file-native retrieval is not a universal best practice; its efficacy is strictly model-dependent.
Format Insights: Identified that while aggregate accuracy is format-agnostic, individual models (especially open-source) exhibit significant format sensitivities.
Scalability Proof: Validated that file-native agents can navigate 10,000-table schemas with high accuracy using domain-partitioned schemas.
The "Grep Tax" Phenomenon: Discovered that compact or novel formats (like TOON) can incur higher runtime token costs due to output density and model unfamiliarity with search patterns, contradicting the assumption that smaller file size equals lower cost.

4. Key Results

A. Architecture Effectiveness (R1)

Frontier Models: File-native retrieval significantly improved accuracy by +2.7% ( $p=0.029$ ) compared to prompt engineering.
Open Source Models: File-native retrieval showed mixed to negative results. Aggregate accuracy dropped by -7.7% ( $p<0.001$ $p < 0.001$ ).
- Drivers: Qwen (-21.9%) and Llama Maverick (-13.9%) performed significantly worse with file agents, while Kimi and Llama Scout showed negligible differences.
Conclusion: Architecture choice must be tailored to model capability.

B. Format Impact on Accuracy (R2)

Aggregate: No significant difference in accuracy across formats (Chi-squared $p=0.484$ ).
Individual Sensitivity: Open-source models showed large performance spreads (9.8%–20.1%) between formats, whereas Frontier models were stable (1.6%–5.4%).
Preference: YAML was best for 5 models, Markdown for 4, JSON for 2, and TOON for 0.

C. Model Capability Dominance (R3)

The Gap: Model tier is the dominant factor. Frontier models achieved 86.0% accuracy, while Open Source models achieved 64.6% (a 21 percentage point gap).
Complexity: All models performed well on simple queries (L1), but Frontier models maintained 64% accuracy on complex reasoning (L5), while Open Source models dropped to 27%.

D. Scaling to 10,000 Tables (R4)

Single-File: Accuracy remained near-perfect up to 1,000 tables.
Partitioned: Using domain-based partitioning (~250 tables per domain), agents maintained high navigation accuracy at 10,000 tables. The per-query context remained bounded regardless of total schema size.

E. Efficiency and the "Grep Tax" (R5)

Token Efficiency: Contrary to file size, YAML was the most token-efficient format (12,729 tokens avg).
- Markdown was +60% more expensive due to verbose pipe-table formatting.
- TOON was +38% more expensive than YAML despite being 25% smaller in file size.
The Grep Tax Mechanism:
1. Token Density: Compact formats pack more content per line, increasing the token count of a single grep hit.
2. Pattern Unfamiliarity: Models unfamiliar with custom keywords (e.g., TOON's TABLE, COL) cycle through incorrect search patterns (e.g., trying DDL or JSON syntax) before finding the right one, increasing tool calls and token usage.
- Example: In one case, a model required 16 grep attempts for TOON vs. 3 for YAML.

5. Significance and Practical Recommendations

This paper shifts the paradigm from "one-size-fits-all" context engineering to model-aware strategies.

Architecture Selection:
- Frontier Models: Use File Agents (on-demand retrieval).
- Open Source Models: Use Prompt Engineering (embed schema) to avoid tool-use inefficiencies.
Format Selection:
- Prioritize YAML for token efficiency and grep-friendliness.
- Avoid novel/compact formats (like TOON) unless specific prompt guidance is provided to mitigate the "grep tax."
Scaling Strategy:
- For enterprise-scale schemas (>1,000 tables), implement domain-partitioned schemas to keep context bounded.
Design Principle:
- Invest in model capability first; format optimization yields diminishing returns compared to choosing the right model tier.
- If using compact formats, provide explicit grep pattern examples in the system prompt to prevent pattern cycling.

Conclusion: The study provides evidence-based guidance for deploying LLM agents, proving that architectural decisions must be tailored to specific model capabilities rather than assuming universal best practices.