🤖 machine learning

Context by Distinct Information: An Auditable Dirichlet-Process Working Memory for Long, Redundant Context Streams

This paper proposes an auditable working memory architecture that organizes context by task dependency—allocating recall, summary, and locality information to distinct components—thereby enabling memory to scale with the number of distinct information items rather than total token count, which improves efficiency and interpretability in long, redundant streams.

Original authors: Siddharth Pal, Viktoria Rojkova

Published 2026-07-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Siddharth Pal, Viktoria Rojkova

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to remember a long, chaotic story. Maybe it's a transcript of a conversation, a log of computer errors, or a list of medical codes. The standard way AI handles this is like a student cramming for a test: they try to memorize every single word in the order it was spoken. If the story is 10,000 words long, the student's brain gets stuffed with 10,000 tiny notes. Even if the story just repeats the same three jokes over and over, the student still writes down every single "haha" and "lol."

This paper suggests a smarter way: Don't memorize the words; memorize the ideas.

The "New Idea" Rule

The authors propose a system that acts like a very picky librarian. Instead of filing a new book for every time someone mentions "pizza," the librarian checks their shelves first.

If "pizza" is already on the shelf, the librarian ignores the new mention and just updates the usage count.
If "pizza" is new (a "novel" item), then the librarian creates a brand new slot for it.

This is called a Dirichlet-Process Working Memory. In plain English, it's a memory that only grows when it encounters something it hasn't seen before. If a stream of data is full of repeats (redundant), this memory stays small. If the data is full of unique facts, the memory grows.

The Three Types of Memories

The paper argues that one size does not fit all. Depending on the task, you need three different kinds of memory buckets:

The "Recency" Window (The Short-Term Buffer):
- What it is: A sliding window that only remembers the last few seconds or words.
- When to use it: When the answer depends on what happened right now.
- The finding: On short tasks (like predicting the next character in a sentence), this simple window is actually better than the fancy new system. The paper explicitly rules out the idea that the new system is a "universal winner" for everything.
The "Summary" Stream (The Recurrent State):
- What it is: A compressed, running summary of the whole story, like a weather report that says "it's been raining all week."
- When to use it: When the answer depends on the average or the trend over a long time.
- The finding: When predicting the cost of a medical claim, the "summary" memory wins. The "new idea" cache (the picky librarian) actually does nothing helpful here because the task doesn't need to recall specific past items; it just needs the general vibe.
The "Distinct Item" Cache (The Picky Librarian):
- What it is: The novelty-gated system that only stores unique items.
- When to use it: When the answer depends on remembering a specific past event, like "What was the diagnosis code for Patient X three months ago?"
- The finding: This is where the magic happens. On tasks like predicting the next medical code or finding a place you visited 500 frames ago, this system beats the standard "memorize everything" approach.

The Big Wins (and the Limits)

The "Half the Work" Result:
In experiments with text (using a dataset called enwik8), the new system managed to predict the next character just as well as the standard system, but it only paid attention to about 49% to 53% of the tokens. It skipped the repeats.

The catch: The paper notes that while the reading is faster and cheaper, the writing (checking if a new item is truly new) is slower. It's a trade-off: you spend more time organizing the library so you can read from it faster later.

The "Long Story" Advantage:
As the story gets longer, the advantage grows.

At a length of 256 tokens, the new system was slightly worse than the old one.
At 512 tokens, they were about the same.
At 1,024 tokens, the new system was clearly better, beating the standard system by 0.300 bits per character.
The paper suggests that for very long contexts, skipping the repeats is a game-changer.

The "Real World" Test:
The authors tested this on real data, like hospital records (MIMIC-IV) and insurance claims (DE-SynPUF).

On a task to predict the next medical code, the new system beat the standard "sliding window" by a significant margin (about 0.311 bits per event at a 1,024-event horizon).
However, on a task to predict the cost of a claim, the new system was neutral. It didn't help, and it didn't hurt. The "summary" memory was the hero there. This proves the paper's main point: you must match the memory type to the task.

What the Paper Says "No" To

It's important to know what this paper doesn't claim:

It is not a magic bullet for everything. The authors explicitly state that for short, local tasks, a simple sliding window is still the best choice.
It doesn't solve "Distribution Shift." In one experiment with computer logs (BGL), the system failed completely. Why? Because the pattern of errors changed over time (the "template distribution drifted"). The system couldn't adapt to the new reality. The paper admits this is a hard wall: if the rules of the game change, the memory gets confused.
It's not ready for complex human conversations yet. The paper does not claim this works for multi-turn chatbots or retrieval agents where you need to understand nuance, contradictions, or who said what. It's a "primitive" (a building block), not a finished product.

The Bottom Line

The paper suggests that we should stop treating context as a long, boring list of tokens. Instead, we should treat it as a collection of distinct items.

If you need to remember a specific fact from the past, use a Novelty Cache (the picky librarian).
If you need to know the general trend, use a Summary (the weather report).
If you just need the last few seconds, use a Window (the short-term buffer).

The authors measured this on public data and found that by mixing these tools, you can build a memory that is auditable (you can see exactly what it remembered) and efficient (it scales with the number of unique things, not the total number of words). But they are careful to say: this is a start, not the finish line. The system works great when the data is repetitive and stable, but it struggles when the data changes its mind.

Technical Summary: An Auditable Dirichlet-Process Working Memory for Long, Redundant Context Streams

1. The Problem: Token-Centric vs. Information-Centric Context

Current context engineering designs treat the token as the fundamental unit of memory. Whether through bounded recurrent states, full key–value (KV) caches, or fixed-window eviction policies, these systems scale memory and compute linearly or quadratically with the number of tokens. This approach is inefficient for long, redundant streams where the same information (e.g., a name, a tool status, a log template, or a code) recurs frequently. In such scenarios, the task often depends on the distinct information carried by the stream rather than the raw token count.

The paper argues that metering context by tokens charges repeatedly for repeated evidence. Instead, the authors propose a context engineering primitive where memory scales with the number of distinct items (novelty classes) rather than the number of observed tokens.

2. Methodology: A Decomposed Working Memory

The paper introduces a working memory architecture that decomposes context based on how a task depends on the past, assigning each dependency type to a specific memory structure:

Recall-Carried Information: Information where a later query depends on a specific earlier item (e.g., a rare code or a specific entity). This is handled by a content-addressed novelty cache.
Summary-Carried Information: Information where the target depends on a long-range aggregate or smooth history. This is handled by a recurrent state (e.g., a state-space model).
Locality-Carried Information: Information where the target depends mainly on the immediate past. This is handled by a recency window.

The Novelty-Gated Cache

The core mechanism is an allocate-on-novelty cache, derived from a companion mechanism paper (Pal and Rojkova, 2026). It operates as follows:

Allocation Rule: For an incoming key–value pair $(k_t, v_t)$ , novelty is calculated as $nov_t = 1 - \max_{i} \text{sim}(k_t, \kappa_i)$ , where $\kappa_i$ are existing slot keys.
Decision: If $nov_t > \tau$ (a threshold), a new slot is appended. Otherwise, the incoming pair is merged into the most similar existing slot.
Read Operation: Queries attend only over the occupied slots, not the full token stream.
Theoretical Basis: This rule corresponds to the small-variance maximum-a-posteriori limit of a Dirichlet-process mixture (DP-means), acting as a differentiable memory operator.

Architecture

The sequence model is a standard residual stack. The mixing operator (read) within each block can be:

Diagonal state-space convolution (fixed-state).
Full causal attention (growing token cache).
Novelty-gated causal attention (content-addressed cache).
Coupled Reads: The proposed component couples a recurrent path (integrating history) with the novelty-gated cache (recalling distinct items).

3. Key Contributions

Context Decomposition: The paper establishes that context is not monolithic. It separates recall-carried, summary-carried, and locality-carried information, demonstrating that a single memory policy (e.g., a sliding window or full attention) is suboptimal for mixed dependencies.
Auditable Memory: Unlike opaque recurrent states or fixed-budget eviction caches, the novelty-gated cache maintains an inspectable table of retained items. In structured domains, slots align with explicit objects (templates, drugs, codes, places), allowing the system to report what was retained and why (via merge provenance).
End-to-End Validation: The paper moves beyond mechanism-level proofs to train models end-to-end, showing that the novelty-gated cache can be successfully coupled with recurrent summaries to match or exceed full-attention performance at a fraction of the cost.

4. Experimental Results

A. Mechanism-Level Evidence (Real Streams)

On four public, ground-truthed streams, the bare allocate-on-novelty mechanism successfully compressed data into distinct-item representations:

System Logs (Loghub HDFS): Compressed 2,000 lines into 15 templates (143× redundancy), recovering ground-truth templates with 0.89 accuracy.
Clinical (MIMIC-IV): Reduced 18,087 prescription events to 628 slots (29× redundancy), matching attention memory performance at 1/29th the size.
Claims (DE-SynPUF): Reduced 150,000 diagnosis events to 3,933 slots (38× redundancy), achieving 0.989 recall of ICD-9 chapters. Fixed-budget eviction failed (0.30 recall).
Mapping (KITTI): Maintained exactly one slot per place, achieving 1.00 revisit recall, whereas sliding windows and heavy-hitter policies failed on long trajectories.

B. Character-Level Control (enwik8)

Novelty vs. Full Attention: Novelty-gated attention achieved parity with full attention (2.073 vs. 2.087 bits per character) while attending to only 49% of tokens.
The Coupled Component: Combining state-space recurrence with the novelty-gated cache achieved 1.877 bpc (attending to 53% of tokens), matching the performance of state-space + full attention (1.932 bpc) but with significantly reduced cache size.
Signal vs. Approximation: Adding the gated cache beside full attention further improved performance (1.836 bpc), proving the cache supplies unique signal rather than merely approximating attention.

C. Context Scaling (Horizon Sweep)

As sequence length increased from 256 to 1,024 tokens, full attention performance degraded significantly (2.36 to 2.69 bpc).
The coupled component remained nearly flat (2.42 to 2.39 bpc), widening the performance gap to +0.300 bpc at 1,024 tokens.
The admitted fraction decreased as context lengthened (0.45 to 0.42), indicating increasing redundancy exploitation.

D. Domain-Specific Findings

Next-Code Prediction (Claims): The coupled component led full attention by ~0.3 bits per event at a 1,024-event horizon. Recency and popularity-based eviction policies failed, confirming that rare, distinct codes drive this task.
Cost Forecasting (Claims): This task was summary-carried. Recurrent reads outperformed attention and the cache. The cache was neutral, demonstrating that the cache does not universally help; the memory policy must match the task's dependency.
Log Streams: The cache worked well on stationary HDFS logs but failed on BGL logs due to distribution shift (template drift), scoping the claim to streams with stable distributions or requiring adaptive budgets.
Keep-First vs. Keep-Latest: For text, keeping the latest occurrence of a repeated key (Keep-Latest) outperformed Keep-First. However, for structured claims, Keep-First was superior, reinforcing that the optimal retention strategy is domain-dependent.

5. Significance and Claims

The paper claims modestly but definitively that context can scale with the distinct information a task needs rather than with tokens.

Architectural Primitive: The novelty-gated cache is a viable, content-addressable, and auditable working memory component.
Conditional Superiority: The advantage of novelty-gated memory grows with context length and redundancy. However, it is not a universal replacement; it is most effective for recall-carried tasks. For summary-carried tasks, recurrence is superior, and for locality-carried tasks, a sliding window is preferable.
Audibility: The retained memory is an inspectable table of templates, codes, or entities, offering transparency that opaque state-space models cannot provide.
Limitations: The experiments are small-scale, use public data, and do not yet evaluate multi-turn assistants or retrieval-augmented agents. The current cosine-based gate and merge rules are not claimed to be sufficient for complex semantic facts (e.g., resolving contradictions or temporal updates) without further frontend engineering.

The central engineering takeaway is that a practical long-context system should not force all information through a single policy. Instead, it should decompose context into distinct-item recall, recurrent summary, and recency windows, coupling them based on the specific demands of the task.