💻 computer science

Reuse, Don't Recompute: Efficient Large Reasoning Model Inference via Memory Orchestration

The paper introduces ENGRAM-R, an inference-time memory orchestration system that significantly reduces token consumption and latency in large reasoning models by retrieving and reusing structured facts instead of recomputing derivations, while maintaining or improving accuracy on complex benchmarks.

Original authors: Daivik Patel, Shrenik Patel

Published 2026-03-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Daivik Patel, Shrenik Patel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Over-Thinker" AI

Imagine you have a brilliant, hyper-intelligent assistant (a Large Reasoning Model, or LRM). This assistant is great at solving hard problems, but it has a bad habit: it never stops talking.

If you ask it a question about something you discussed last week, instead of just remembering the answer, it decides to re-read the entire conversation from the beginning, re-explain the context, and then re-derive the answer from scratch.

The Cost: This takes a huge amount of time (latency) and costs a lot of money (tokens). It's like asking a chef to re-grow the vegetables from the seed every time you order a salad, rather than just grabbing them from the fridge.
The Result: The AI is accurate, but it's slow, expensive, and inefficient.

The Solution: The "Smart Librarian" (ENGRAM-R)

The authors of this paper, Daivik and Shrenik Patel, propose a new system called ENGRAM-R. They argue that the AI shouldn't "think less" by being dumber; it should "think less" by reusing what it already knows.

Think of ENGRAM-R as a Smart Librarian who sits between you and the AI. Here is how it works, step-by-step:

1. The "Filing System" (Memory Orchestration)

Instead of letting the AI drown in a sea of text, the Librarian organizes your past conversations into three neat, labeled boxes:

The Diary Box (Episodic): "What happened and when?" (e.g., "We moved to Seattle last year.")
The Fact Book (Semantic): "What is true?" (e.g., "My favorite color is green.")
The Instruction Manual (Procedural): "How do we do things?" (e.g., "File taxes by April 15.")

2. The "Flashcard" Transformation (Fact Cards)

When you ask a question, the Librarian doesn't hand the AI a 50-page transcript of your old chat. That would be overwhelming.
Instead, the Librarian reads the relevant parts and turns them into tiny, atomic Flashcards.

Old Way: "Remember that long paragraph where we talked about moving to Seattle? Well, it said..." (AI has to read and re-explain).
ENGRAM-R Way: The Librarian hands the AI a card that says: [E1] A moved to Seattle.

3. The "Citation Rule" (No Re-telling)

This is the secret sauce. The Librarian gives the AI a strict rule: "You must answer using these flashcards, and you must point to the card number. Do not rewrite the story."

The AI's Job: Instead of writing a paragraph explaining the move, it just writes: "A lives in Seattle [E1]."
The Result: The AI skips the "thinking" part of re-deriving facts it already has. It just connects the dots.

Why This Changes Everything

The paper tested this system on two huge challenges:

LoCoMo: A benchmark with very long, complex conversations.
LongMemEval: A test of remembering details over long periods.

The Results were shocking:

85% Less Input: The AI had to read 85% less text because the Librarian gave it the "Flashcards" instead of the "Novel."
75% Less Thinking: The AI generated 75% fewer words in its reasoning process because it didn't have to re-explain facts.
Faster & Cheaper: The answers came much faster and cost much less to produce.
Better Accuracy: Surprisingly, the AI was more accurate on complex, multi-step questions because it wasn't getting lost in a sea of words.

The Analogy: The Chef vs. The Sous-Chef

The Old Way (Full Context): You are a Chef (the AI). Every time you need to make a sandwich, you have to walk to the garden, pick the lettuce, wash it, chop it, and then make the sandwich. You do this even if you just made a sandwich 5 minutes ago.
The New Way (ENGRAM-R): You have a Sous-Chef (the Memory Layer). When you ask for a sandwich, the Sous-Chef says, "I already washed and chopped the lettuce. Here is a prepped tray with the ingredients ready to go. Just assemble it."
- You save time.
- You save energy.
- The sandwich tastes the same (or better, because you didn't get tired).

The Bottom Line

The paper's main message is: "Reuse, Don't Recompute."

We don't need AI to be smarter by thinking longer; we need AI to be smarter by remembering better. By organizing memory into compact, citable "Flashcards," we can make powerful AI systems that are faster, cheaper, and just as accurate as the current giants. It's the difference between re-reading a whole book to find a quote versus just looking it up in the index.

1. Problem Statement

Large Reasoning Models (LRMs) currently achieve high accuracy through Test-Time Scaling (TTS), which involves generating long chains of thought (CoT) or sampling multiple solutions. While effective, this approach incurs steep costs:

Token Inefficiency: Models often re-compute derivations and re-narrate evidence they have already "seen" in the context window.
Latency & Cost: Long reasoning chains increase inference time and token consumption, making deployment in resource-constrained environments (e.g., edge devices, high-throughput systems) impractical.
Context Limits: Naive approaches that feed entire conversation histories into the model exceed context windows and dilute attention mechanisms.

The authors argue that the solution is not "thinking longer," but thinking less by reusing structured, pre-computed memory rather than recomputing derivations.

2. Methodology: ENGRAM-R

The paper introduces ENGRAM-R, an inference-time memory orchestration layer that sits outside the model weights. It transforms the standard retrieval-augmented generation (RAG) pipeline into an efficiency engine.

Core Architecture

The system extends the existing ENGRAM memory architecture with two critical innovations:

Typed Memory Stores: Dialogue turns are routed into three distinct memory types:
- Episodic: Specific events and interactions.
- Semantic: Facts, entities, and static knowledge.
- Procedural: Routines, rules, and workflows.
- Mechanism: A lightweight router classifies incoming utterances, normalizes them into records, and stores them with embeddings in a relational database.
Fact Card Rendering (The Key Innovation):
- Instead of retrieving raw, verbose text snippets, the system re-renders retrieved records into compact Fact Cards.
- A Fact Card $\phi(m)$ $ϕ (m)$ consists of:
  - ID: A stable identifier (e.g., [E1]).
  - Claim: A minimal, canonical statement distilled from the record.
  - Anchor: A timestamp or provenance marker.
- This compresses multi-sentence evidence into atomic, verifiable units.
Controlled Citation Mechanism:
- The prompt construction forces the LRM to answer based only on the set of Fact Cards $F(q)$ .
- Constraint: The model must justify its answers by explicitly citing Fact Card IDs (e.g., "A lives in Seattle [E1]") rather than paraphrasing the evidence.
- Effect: This prevents the model from "re-narrating" evidence, drastically shortening the reasoning chain.

Inference Pipeline

Query Time: The system retrieves top- $k$ candidates from each typed store using dense similarity search.
Aggregation: Candidates are merged, deduplicated, and truncated to a fixed budget (typically $K=25$ ).
Transformation: Retrieved records are converted into Fact Cards.
Prompting: The LRM receives the query and the Fact Cards, with instructions to cite IDs.
Generation: The model produces a concise answer with explicit citations.

3. Key Contributions

ENGRAM-R System: A model-agnostic inference layer that reduces test-time compute without fine-tuning model weights.
Fact Card Abstraction: A novel representation that converts verbose retrieval results into compact, auditable evidence units.
Controlled Citation: A prompting strategy that enforces direct referencing of memory, eliminating redundant reasoning steps.
Empirical Validation: Demonstrated on long-horizon benchmarks that memory orchestration can replace large fractions of CoT while maintaining or improving accuracy.

4. Results

The authors evaluated ENGRAM-R on LoCoMo (multi-session dialogues) and LongMemEvalS (extended user-assistant histories), using gpt-oss-20b as the backbone.

Efficiency Gains

Input Tokens: Reduced by ~85% (LoCoMo) and ~96% (LongMemEvalS) compared to full-context baselines.
Reasoning Tokens: Reduced by ~75% (LoCoMo) and ~78% (LongMemEvalS).
Latency: Median end-to-end latency dropped by ~68% (LoCoMo) and ~81% (LongMemEvalS).

Accuracy Performance

Multi-hop & Temporal Reasoning: Accuracy improved significantly.
- LoCoMo Multi-hop: +2.5% accuracy.
- LoCoMo Temporal: +1.9% accuracy.
- LongMemEvalS Multi-session: +30.1% accuracy (from 36.8% to 66.9%).
Mechanism: By providing structured, non-redundant evidence, the model avoids getting "lost" in long contexts and focuses on logical composition.
Trade-offs: Slight accuracy dips were observed in "Open-domain" and "Single-hop" categories where broad context or immediate local context is preferred over strict retrieval, though the efficiency gains remained substantial.

Ablation Study

Comparing ENGRAM Base (standard retrieval) vs. ENGRAM-R (Fact Cards + Citation):

The addition of Fact Cards and citation enforcement reduced reasoning tokens by nearly 50% overall.
It specifically boosted performance on complex tasks (Multi-hop +7.9%, Temporal +7.7%) by forcing the model to rely on distilled facts rather than re-deriving them.

5. Significance and Implications

Paradigm Shift: Challenges the prevailing "scale compute to get better answers" narrative. It posits that reuse of structured memory is a more efficient path to reliability than recompute.
Deployment Viability: Makes LRMs viable for resource-constrained environments (e.g., edge devices, NGOs, education) by drastically lowering token budgets and latency.
Auditability: The explicit citation mechanism creates a transparent reasoning trace where every claim can be traced back to a specific, compact Fact Card, enhancing trust and debuggability.
Generalizability: The approach is model-agnostic and can be applied to any LRM, offering a practical lever for efficiency without requiring architectural changes to the model itself.

In conclusion, ENGRAM-R demonstrates that by treating memory as a structured, typed, and citable resource, Large Reasoning Models can achieve high-fidelity long-horizon reasoning with a fraction of the computational cost.