🤖 AI

The Hidden Footprint: Making Storage a First-Class Metric for LLM Agent Evaluation

This paper introduces AgentFootprint, a benchmark revealing that LLM agents' persistent storage footprints vary drastically across frameworks and configurations independent of task accuracy, while demonstrating that content-addressed storage can significantly reduce this overhead without compromising data reconstructability.

Original authors: Chenglin Yu, Hongquan Gui, Ying Yu, Hongxia Yang, Ming Li

Published 2026-07-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Chenglin Yu, Hongquan Gui, Ying Yu, Hongxia Yang, Ming Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're watching a brilliant robot detective solve a mystery. It reads clues, asks questions, and writes down its final answer. Everyone cheers when the detective gets the right answer, and they check how fast it thought and how much it cost to run. But nobody is looking at the mess the detective leaves behind on the floor.

This paper, called The Hidden Footprint, is like a janitor's report. It asks: "When the detective finishes, how much trash does it leave in the room?"

The Big Surprise: The Mess is Huge

The main finding is shocking: Two detectives can solve the exact same mystery perfectly, but one leaves behind a pile of trash 15.7 times bigger than the other.

Think of it like this: You and your friend both bake a perfect chocolate cake. You hand over the cake (the result). But while you just wiped the counter, your friend left behind 15 bowls, 30 cups of flour, and a mountain of sticky wrappers. The cake looks the same, but your friend's kitchen is a disaster zone.

In the real world, this "trash" isn't just paper; it's digital bytes stored on a hard drive. The researchers found that for a single task, some frameworks leave behind 131 MB of data, while the actual answer (the "delivered cake") was only 2.6 MB. That means the framework's "residue" was 24,033 times bigger than the actual work it was supposed to do!

The "Invisible" Double-Counting Trick

Here is the tricky part: If you just count the files on the computer, you might think the mess isn't that bad. The paper argues that standard counting is a trap.

Imagine you write the word "Hello" on a piece of paper. Then, you photocopy that paper, but the photocopier adds a tiny "invisible ink" stamp to every copy. If you count the paper, you see one sheet. But if you look at the ink, you see the word "Hello" written twelve times.

The researchers found that because of how computers save data (using things like "SQLite" pages and "JSON" escaping), the same information gets stored over and over again, hidden inside the system.

Naive counting said: "Oh, there's only a little duplication."
The paper's smart counting said: "Actually, the same content is stored 12.1 times!"

They proved that if you don't look inside the "invisible ink," you miss the real size of the mess by a huge margin.

The "Growth" Problem

The paper also tested what happens when the detective has to check the same clue 200 times.

Some frameworks (like LangGraph and AutoGen) act like a squirrel that never forgets anything. Every time it checks the clue, it writes down the entire history again. This causes the storage to grow superlinearly (faster and faster). After 200 rounds, they left behind 323 MB of data just for one tiny file.
Other frameworks (like OpenAI Agents) act like a smart note-taker. They only write down the new stuff. Their storage grew slowly, almost in a straight line.

The paper measured this growth rate (called $\alpha$ ) and found it ranged from 0.73 (getting smaller!) to 1.95 (exploding in size).

Does a Bigger Mess Mean a Smarter Detective?

You might think, "Maybe the messy detective is just being extra careful, so it's smarter?"
The paper says: No.

They looked at 108 real-world submissions from a famous coding challenge (SWE-bench). They found that the amount of data exported by these systems varied by 1,617 times (from tiny files to huge ones). But here is the kicker: There was no connection between the size of the mess and how well the system solved the problem.

In fact, the correlation was so weak it was basically zero. A system could leave behind a mountain of data and still fail, or leave behind a tiny footprint and succeed. The paper explicitly rules out the idea that "more storage = better performance."

The "Magic Eraser" (But Don't Buy It Yet)

The researchers didn't just point out the problem; they showed a way to fix it, but they are careful to say this is a proof of concept, not a finished product.

They built a "content-addressed store." Imagine a library where, instead of putting every book on a shelf, you take a photo of the book's unique fingerprint. If two books are identical, you only keep one photo and a note saying "This one is the same as that one."

When they applied this "magic eraser" to the messy frameworks:

It reduced the storage by 4.8 to 32.7 times.
Crucially, they proved that even with all that data removed, you could still perfectly reconstruct the entire conversation history. The "score" for being able to replay the detective's thoughts stayed exactly the same.

What the Paper Rules Out

It rules out the idea that we should just ignore storage because "it's cheap." The paper argues that storage is a persistent debt that accumulates over time, unlike the cost of running the AI which disappears when the task ends.
It rules out the idea that standard file counters are accurate. They showed that simple counting misses the duplication hidden inside the system's code.
It rules out the idea that bigger data trails mean better results. The data shows no link between size and success.

How Sure Are They?

The authors are very sure about the measurements they took. They ran 1,061 experiments in isolated, clean computer rooms (sandboxes) to make sure nothing else interfered. They tested 8 different frameworks (like LangGraph, CrewAI, and AutoGen) using the exact same tasks and models.

They didn't just guess; they measured. They found that under identical conditions, the amount of data left behind varied by 15.7 times between the best and worst performers. They also showed that this difference isn't because one framework is "smarter" at solving tasks, but because of how they are built to save data.

The Takeaway

The paper concludes that we need to start measuring the "footprint" of AI agents just like we measure their speed or cost. Right now, we are letting some systems leave behind mountains of digital trash while others leave almost nothing, and it turns out the messy ones aren't doing a better job.

The authors suggest that if we want to run these agents at a massive scale (like a fleet of 10,000 robots a day), the difference between a "clean" framework and a "messy" one could mean the difference between needing 3 GB of storage or 51 GB every single day. That's a huge difference for a problem that, as they showed, doesn't even make the robot smarter.

Technical Summary: The Hidden Footprint: Making Storage a First-Class Metric for LLM Agent Evaluation

1. Problem Statement

Current benchmarks for Large Language Model (LLM) agents focus on task completion, reliability (e.g., pass@k), and inference cost. However, they systematically ignore the persistent storage footprint—the bytes an agent execution leaves on disk after completion. This includes logs, context snapshots, checkpoints, debug traces, and session databases.

The paper argues that this omission is critical because:

Persistence vs. Flow: Unlike inference cost (a flow consumed during execution), storage is a stock that accumulates across runs, impacting long-term storage costs, data retention compliance, and the ability to replay or audit agent behavior.
Underestimation: In production deployments, framework residue can exceed delivered artifacts by orders of magnitude (e.g., 79 MB of residue vs. 2.6 MB of output).
Measurement Traps: Naïve byte-level measurements (e.g., du -sh) fail to detect logical duplication because serialization mechanisms like SQLite paging and JSON escaping obscure repeated content. For instance, a store logically holding the same content 12 times might report a duplication factor ( $D$ ) of only 1.01 when measured at the raw file level.

2. Methodology: AgentFootprint

The authors introduce AgentFootprint, a systematic cross-framework benchmark designed to measure post-run storage footprints.

2.1 Experimental Setup

Frameworks: Eight representative frameworks were evaluated: LangGraph, AutoGen, CrewAI, SmolAgents, OpenAI Agents SDK, LlamaIndex, Agno, and InfiAgent.
Control: All frameworks were run under identical conditions: the same backend model (DeepSeek-V4-Flash, temperature 0), identical tools, and documented "durable-session" configurations (no storage-specific tuning).
Isolation: Each run occurred in a fresh sandbox (Docker containers) with workspace and home directories snapshotted before and after execution to isolate the delta.
Task Suites:
- File-QA: 10 tasks involving re-reading documents to test read amplification.
- Long-Horizon: 25–200 rounds of re-reading a 2 KB status file to measure growth exponents.
- Write-Task: Generating summary files to compare legitimate artifacts vs. framework residue.
- Exploratory/Heterogeneous: Tasks graded on both answer correctness and artifact delivery.
- Fixed-Trace Control: Replaying an identical trajectory through different adapters to isolate the persistence layer's amplification from agent behavior.

2.2 Metric Suite

The benchmark defines six metrics to address serialization masking and semantic ambiguity:

$S_{total}$ : Total retained bytes (workspace artifacts + framework residue).
Store Composition: Breakdown by channel (state/checkpoints, debug logs, conversation history, etc.).
$D$ (Duplication Factor): $S_{logical}^{total} / S_{unique}$ . Calculated by analyzing logical content streams (SQLite cells, JSONL lines) rather than raw files, using content-defined chunking (FastCDC) and content probes to detect JSON-escaped duplicates.
$\alpha$ (Growth Exponent): Fitted from $\log S_{total} \sim \alpha \log T$ over repeated-observation tasks to determine if storage grows linearly, sublinearly, or superlinearly.
$C$ (Compressibility): Ratio of $S_{total}$ to its size under zstd compression (a syntactic baseline).
$R$ (Reconstructability Score): A 0–3 score indicating whether retained bytes allow for the exact reconstruction of the conversation history at step $k$ .

3. Key Results

3.1 Equal Accuracy, Massive Dispersion

15.7× Difference: At identical 100% task accuracy, retained bytes varied by a factor of 15.7 between frameworks (LangGraph: 5.10 MB vs. CrewAI: 0.32 MB per task).
Zero Retention: SmolAgents retained 0 bytes, achieving 100% accuracy but offering no replay or recovery capability ( $R=0$ ).
Residue Dominance: In write tasks, framework residue exceeded legitimate output files by up to 24,033×.

3.2 Growth Regimes

Superlinear Growth: Frameworks persisting full history at every step (LangGraph, AutoGen, LlamaIndex) exhibited growth exponents ( $\alpha$ ) between 1.74 and 1.95. A 200-round monitoring loop left 323 MB of data.
Linear/Sublinear Growth: Frameworks with windowed context or bounded stores (InfiAgent, Agno, CrewAI) stayed near-linear ( $\alpha \approx 0.98–1.20$ ).
Fixed-Trace Amplification: When replaying an identical trajectory, the persistence layer alone caused a 6.7× difference in retained size across frameworks.

3.3 Serialization Masks Duplication

Raw file chunking reported a duplication factor ( $D$ ) of 1.01 for LangGraph.
Logical stream analysis revealed $D = 12.8$ , indicating that serialization (SQLite paging, JSON escaping) obscures massive redundancy.
Naïve measurements understate duplication by an order of magnitude.

3.4 In-the-Wild Findings (SWE-bench)

Analysis of 108 instance-normalized SWE-bench Verified submissions showed a 1,617× spread in exported trajectory volume.
No Correlation: There was no detectable correlation between trajectory volume and resolve rate ( $\tau_b = 0.080, p = 0.222$ ). Heavier trajectories did not correlate with higher success.

3.5 Mitigation Potential

A content-addressed store (CAS) compactor reduced retention by 4.8–32.7× across frameworks while preserving all reconstructability scores ( $R$ ) and passing restore checks.
This demonstrates that the "waste" is removable redundancy, not a fundamental requirement for high-fidelity history reconstruction.

4. Contributions

Metric Suite & Methodology: A novel set of metrics ( $S_{total}, D, \alpha, C, R$ ) and a measurement methodology that accounts for serialization masking, revealing that naïve byte counts significantly understate duplication.
AgentFootprint Benchmark: A reproducible harness executing 1,061 sandboxed runs across eight frameworks, providing the first systematic cross-framework comparison of post-run storage.
In-the-Wild Study: Evidence from SWE-bench showing that storage volume is uncorrelated with success and varies wildly even among successful submissions.
Existence Proof: A demonstration that a content-addressed store can reduce storage by up to 32.7× without sacrificing history reconstructability, proving that current default configurations are far from the theoretical minimum.

5. Significance and Claims

The paper claims that storage footprint is a critical, previously unmeasured dimension of agent evaluation. It argues that:

Storage is a System-Level Resource: Like memory footprint for mobile apps, unmeasured storage constraints deployment at scale (e.g., fleet operations accumulating terabytes daily).
Defaults are Not Efficient: Documented "durable" configurations often prioritize recovery features over efficiency, leading to massive, unnecessary storage accumulation.
Accuracy is Insufficient: Leaderboards focusing solely on task success or inference cost penalize frameworks for low accuracy but ignore frameworks that achieve high accuracy with 15× more storage.
Reconstructability is Achievable at Low Cost: The paper demonstrates that exact history reconstruction does not require megabytes of data; the same $R=3$ score can be achieved with significantly fewer bytes if redundancy is managed.

The authors position AgentFootprint not as a proposal for a new storage system, but as a measurement tool to expose the "hidden footprint" and enable the community to optimize for storage efficiency alongside accuracy and cost.