Contextual Memory Virtualisation: DAG-Based State Management and Structurally Lossless Trimming for LLM Agents

This paper proposes Contextual Memory Virtualisation (CMV), a system that models LLM session history as a version-controlled Directed Acyclic Graph (DAG) to enable state reuse across parallel sessions and employs a three-pass algorithm to structurally losslessly trim mechanical overhead, achieving significant token reduction while preserving all user and assistant content verbatim.

The Gist

Imagine you are a master architect working with a brilliant but slightly forgetful AI assistant. You spend hours together designing a complex skyscraper. You discuss the foundation, choose the materials, debate the lighting, and solve structural problems. By the end of the day, you have built a massive "mental model" of the building in your shared conversation history.

Now, imagine the AI's memory (its "context window") is like a small whiteboard. It's getting full. To make room for new ideas, the AI is forced to erase 98% of what you've written, leaving only a tiny, vague summary like: "We discussed a skyscraper."

When you start working the next day, the AI has forgotten everything. You have to spend another hour re-explaining the foundation, re-debating the materials, and re-solving the problems. This is the current problem with AI agents: they lose their hard-earned understanding every time the conversation gets too long.

This paper proposes a solution called Contextual Memory Virtualisation (CMV). Here is how it works, using simple analogies:

1. The "Git" for Your Brain (The DAG Model)

Currently, talking to an AI is like a single, straight line of text. If you hit a dead end or want to try a different design, you have to start over.

The authors suggest treating your conversation history like software code (specifically using a system called Git).

• Snapshots: Instead of just letting the conversation scroll off the screen, you can take a "snapshot" of the AI's current understanding. Think of this as saving a "checkpoint" in a video game.
• Branching: From that checkpoint, you can create a "branch." Imagine you have a solid foundation for your skyscraper. You can now spawn two parallel universes:
  • Branch A: Focuses on the plumbing.
  • Branch B: Focuses on the electrical wiring.
• The Magic: Both branches start with the exact same deep understanding of the foundation. You don't have to re-explain the foundation to either of them. You can switch between these "parallel sessions" instantly, keeping the AI's brain intact.

2. The "Structural Vacuum Cleaner" (Lossless Trimming)

Even with snapshots, the conversation logs can get huge because they are full of "junk" data that the AI doesn't need to keep in its active memory.

Imagine your conversation log is a suitcase packed for a trip.

• The Good Stuff: Your clothes (the actual conversation, your questions, and the AI's smart answers).
• The Junk: The cardboard boxes the clothes came in, the plastic wrapping, and the shipping labels (raw code dumps, huge base64 images, technical metadata).

The paper introduces a three-pass trimming algorithm that acts like a super-smart vacuum cleaner.

• It keeps your clothes (every user message and AI response) exactly as they are.
• It throws away the cardboard and plastic (raw tool outputs, massive image files, and technical logs).
• The Trick: If the AI needs to see a specific file later, it doesn't need the 50-page dump in the suitcase. It just needs a tiny note saying, "File X was read." The AI can go fetch the fresh file from the hard drive if it really needs to.

This process shrinks the suitcase (the context window) by an average of 20%, and in messy sessions, by up to 86%, without losing any actual meaning.

3. Is It Worth the Cost? (The Economic Check)

You might ask: "If I delete all that junk, won't the AI get confused? And does it cost money?"

• The "Cache" Penalty: AI companies charge less if you send them the same text repeatedly (like a library book they already have). When you trim the text, the AI sees a "new" book and has to pay the full price to read it once.
• The Payoff: The paper ran a test on 76 real coding sessions. They found that even with that one-time "new book" fee, the savings from sending smaller messages every turn paid for themselves very quickly.
  • For sessions with lots of "junk" (like heavy tool usage), the savings kicked in after just 10 turns.
  • For most users, the "break-even" point was around 35 turns.

The Big Picture

Right now, using AI for long-term projects feels like trying to build a cathedral with a memory that resets every hour. You spend 90% of your time re-teaching the AI what it just learned.

CMV changes the game:

1. It saves your progress: You can save your "mental model" of a project and branch off into new ideas without starting from zero.
2. It cleans the house: It removes the digital clutter so the AI can focus on the important stuff.
3. It saves money: It makes long, complex sessions cheaper and faster.

In short, this system treats the AI's memory not as a fleeting chat, but as a version-controlled, persistent workspace that you can manage, branch, and optimize just like a professional software project.

Technical Summary

1. Problem Statement

Large Language Model (LLM) agents engaged in extended reasoning tasks (e.g., complex coding sessions) accumulate significant "state" within their context window. This state includes architectural mappings, trade-off decisions, and codebase conventions.

• The Bottleneck: When the context window reaches its limit, native compaction mechanisms (e.g., Claude Code's /compact) summarize this accumulated state into a few sentences.
• The Consequence: This process discards 98% of the nuanced understanding built over hours of work (e.g., reducing 132k tokens to 2.3k). Consequently, every new session or branch starts from scratch, forcing the model to re-derive context at a high computational and financial cost.
• Limitations of Existing Solutions:
  • RAG: Augments prompts with documents but fails to preserve conversational state.
  • MemGPT: Uses paging but is limited to single sessions and relies on the model to manage memory.
  • Native Utilities: Provide undo/fork capabilities but lack version control, lineage tracking, or efficient context cleanup.
  • Prompt Compression: Methods like LongLLMLingua compress tokens semantically but do not address the structural bloat of raw tool outputs and metadata.

2. Methodology: Contextual Memory Virtualisation (CMV)

The authors propose Contextual Memory Virtualisation (CMV), a system that treats LLM conversational history as version-controlled state, analogous to virtual memory in operating systems. It decouples the cost of building context from the cost of executing a task.

A. The DAG-Based State Model

CMV models session history as a Directed Acyclic Graph (DAG) rather than a linear log.

• Nodes (Snapshots): Immutable copies of a session's JSONL conversation log at a specific point in time, annotated with metadata (timestamp, tags, token count).
• Edges (Branches): Represent independent work sessions forked from a snapshot. A branch receives a copy of the parent's state (optionally trimmed) and a fresh session ID.
• Operations:
  • Snapshot: Freezes the current state.
  • Branch: Creates a new parallel workstream from a snapshot.
  • Trim: Captures, trims, and launches a new session in one step.
  • Tree: Visualizes the lineage (similar to git log --graph).

B. Three-Pass Structurally Lossless Trimming Algorithm

To manage token limits without losing semantic understanding, CMV employs a streaming algorithm that removes "mechanical bloat" while preserving the conversation's core logic.

• Goal: Reduce token count by stripping raw tool outputs, base64 images, and metadata while keeping every user message and assistant response verbatim.
• The Three Passes:
  1. Pass 1 (Boundary Detection): Scans raw lines to detect the last "compaction boundary" (where native summarization occurred). Lines before this boundary are discarded as they are already summarized.
  2. Pass 2 (Orphan ID Collection): Collects IDs of tool use blocks that existed before the compaction boundary. This is crucial for detecting "orphaned" results.
  3. Pass 3 (Stream Processing & Filtering): Applies specific rules to the log:
     • Skip: Pre-boundary content and metadata (file history, queue ops).
     • Strip: Base64 images and non-portable thinking blocks.
     • Stub: Large tool results and write-tool inputs are replaced with a placeholder (e.g., [Trimmed: ~N chars]), preserving metadata fields (file paths, commands) so the model knows what happened without reading the raw data.
     • Orphan Removal: Silently discards tool result blocks whose corresponding tool use IDs were collected in Pass 2 (preventing API validation errors).

3. Key Contributions

1. DAG State Model: A formal framework for conversational memory that enables branching, lineage tracking, and context reuse across parallel sessions, treating context as a persistent resource rather than ephemeral data.
2. Structurally Lossless Trimming: A novel algorithm that reduces token counts by removing structural overhead (tool outputs, images) without altering the semantic content of user/assistant dialogue, ensuring API correctness by handling orphaned tool results.
3. Economic Viability Analysis: A rigorous cost-benefit analysis demonstrating that the savings from reduced token usage outweigh the one-time penalty of invalidating prompt caches.

4. Results

The authors evaluated the system using 76 real-world coding sessions from a single user (Claude Opus 4.6).

• Token Reduction:
  • Mean Reduction: 20% across all sessions.
  • Max Reduction: 86% (in sessions with high mechanical overhead).
  • Median Reduction: 12%.
  • Session Types: "Mixed" sessions (high tool usage) saw a mean reduction of 39%, while purely conversational sessions saw 17%.
• Economic Break-Even:
  • Despite the "cold cache" penalty incurred when the trimmed prompt first loads, the reduced per-turn cost allows users to recover this penalty quickly.
  • Mean Break-even: 35 turns.
  • Mixed Sessions: Reach break-even in just 10 turns.
  • Cost Impact: For a session with 46% reduction, the break-even point was reached at turn 6.
• Context Rebuilding Savings: The paper notes that the primary value is avoiding the cost of rebuilding context from scratch. Branching from a snapshot eliminates the 10–20 turns and 15–30 minutes usually required to re-derive architectural understanding.

5. Significance and Future Work

• Paradigm Shift: CMV moves LLM interaction from a linear, ephemeral model to a version-controlled, persistent state model. It solves the "context window limit" problem at the tooling layer without waiting for model-level breakthroughs.
• Complementarity: It complements existing semantic compression methods by operating on the raw log structure, making it compatible with any agent using tool-call schemas.
• Limitations:
  • Trimming is structural, not semantic. If a stripped tool result is critical for future reasoning, the model might hallucinate or request a re-read (mitigated by preserving the model's own summaries of those tools).
  • Evaluation is limited to a single-user case study.
• Future Directions:
  • Integration into AIOS (LLM Agent Operating Systems) as a first-class persistent context subsystem.
  • Controlled studies to quantify reasoning degradation from structural trimming.
  • Adaptive trimming thresholds based on usage patterns.

Conclusion: The paper demonstrates that by treating context as a version-controlled resource and applying structurally lossless trimming, LLM agents can maintain long-term memory and efficiency across complex, multi-day tasks, making extended reasoning economically and technically viable today.