🤖 AI

Improving Coherence and Persistence in Agentic AI for System Optimization

The paper introduces Engram, an agentic researcher architecture that overcomes the limitations of evolutionary bias and context degradation in system optimization by decoupling long-horizon exploration into iterative agent runs that persist knowledge through a structured Archive and a distilled Research Digest, thereby achieving superior performance across diverse domains.

Original authors: Pantea Karimi, Kimia Noorbakhsh, Mohammad Alizadeh, Hari Balakrishnan

Published 2026-03-24

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Pantea Karimi, Kimia Noorbakhsh, Mohammad Alizadeh, Hari Balakrishnan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a incredibly complex puzzle, like designing the most efficient route for a global delivery trucking company or figuring out the perfect way to organize a massive library. You have a brilliant, super-smart assistant (an AI) who can read millions of books and write code, but this assistant has a few specific personality quirks that make solving new problems hard.

This paper introduces a new way of working with this AI, called Engram, to fix those quirks.

Here is the story of how Engram works, using some simple analogies.

The Problem: Two Ways the AI Gets Stuck

The authors noticed that when they asked standard AI systems to design complex computer systems, they failed in two specific ways:

1. The "Hammer and Nail" Trap (Evolutionary Neighborhood Bias)
Imagine you are trying to fix a broken clock.

The Old Way: You have a robot that tries to tweak the clock by turning one tiny screw, checking if it ticks faster, then turning another screw. It keeps making tiny adjustments.
The Problem: Sometimes, the clock doesn't need a screw turned; it needs the entire mechanism replaced. But because the robot is only looking at tiny screw-turns, it never realizes it needs to change the whole design. It gets stuck in a "local optimum"—it thinks it's doing great because it's making tiny improvements, but it's missing the big picture. It's like trying to fix a flat tire by polishing the rim; it looks shiny, but the car still won't drive.

2. The "Forgetful Genius" (The Coherence Ceiling)
Imagine you hire a genius architect to design a skyscraper.

The Old Way: The architect starts working. They spend weeks drawing blueprints, testing materials, and failing. But because they are working in a single, long conversation with you, they eventually get "brain fog." They forget what they learned in the first week by the time they get to the tenth week.
The Problem: If you start a new architect to finish the job, they know nothing about what the first one learned. They have to rediscover everything from scratch. The knowledge doesn't stick.

The Solution: Engram (The "Research Team")

The authors created Engram, which is like hiring a team of researchers instead of one long-term employee. Instead of one AI working forever, Engram uses a sequence of short-term agents who pass the baton to each other.

Here is how the process works, step-by-step:

1. The "Fresh Start" Rule

Instead of one AI trying to remember everything for months, Engram uses a new AI agent for each "shift."

Analogy: Think of a relay race. Runner A runs for a while, then passes the baton to Runner B. Runner B doesn't need to remember the whole race; they just need to know where Runner A left off.
Why it helps: Every new agent starts with a clean, fresh brain. They don't suffer from "brain fog" or forgetfulness because their context window (their short-term memory) is always fresh.

2. The "Research Digest" (The Secret Sauce)

This is the most important part. When an agent finishes its shift, it doesn't just leave. It writes a Research Digest.

Analogy: Imagine a detective writing a case file before handing the case to the next detective. The file doesn't just say "I tried this and it failed." It says: "I tried locking the door, but the thief climbed the window. So, next time, don't just lock the door; check the windows. Also, the thief is afraid of loud noises."
What's in the Digest: It contains high-level insights, what worked, what failed, and why it failed. It strips away the messy code and logs, keeping only the "wisdom."
The Handoff: The next agent reads this Digest before starting. They immediately know, "Okay, I shouldn't try the door lock again; I should focus on the windows."

3. The "Archive" (The Library)

Along with the Digest, the agent saves all the raw data (code, logs, test results) in a massive Archive.

Analogy: This is like a library. If the next agent wants to see the exact blueprint the previous agent drew, they can go to the library and pull it out. They don't need to carry the whole library in their head; they just pull out what they need when they need it.

Why This is a Big Deal

The paper tested Engram on three very hard problems:

Cloud Multicast: Figuring out the cheapest way to send data to many computers across the world.
LLM Request Routing: Deciding which AI server should handle a user's request to make it fast.
Database Optimization: Organizing data so AI can read it faster.

The Results:

Beating Humans: In the cloud problem, Engram found a solution that was actually better than the best solution created by human experts.
Beating Other AIs: It beat other AI methods that either got stuck in tiny tweaks (the "Hammer and Nail" trap) or forgot what they learned (the "Forgetful Genius").
Tolerating Failure: Because the agents read the Digest, they aren't afraid to try something that looks like it might fail at first. They know that a temporary failure might teach a valuable lesson that leads to a huge win later.

The Takeaway

Engram is a system that treats AI research like a scientific team rather than a solo act. By breaking the work into short shifts and having each researcher write a clear summary of their lessons learned, the system can solve problems that are too complex for a single AI to handle alone. It combines the creativity of a human researcher with the persistence of a team that never forgets a lesson.

In short: Don't let the AI forget what it learned. Write it down, pass it on, and keep moving forward.

1. Problem Statement

Designing high-performance heuristics for complex computer systems is a creative, iterative process requiring experts to form hypotheses, execute multi-step conceptual shifts, and analyze experimental results. While Large Language Models (LLMs) show promise in automating this loop, existing approaches fail to produce expert-level solutions for complex system problems due to two critical failure modes:

Evolutionary Neighborhood Bias: Code-evolution systems (e.g., OpenEvolve, FunSearch) rely on scalar benchmark scores to select code variants. They struggle when improvements require coordinated, multi-step changes (e.g., reformulating a problem, adding relaxations, or accepting temporary performance regressions to switch algorithmic families). They tend to get trapped in local optima (e.g., Steiner-tree heuristics) and cannot navigate non-monotonic progress.
The Coherence Ceiling: Agentic frameworks (e.g., Glia) that use reasoning and tools suffer from "context rot" in long-running single-agent threads. As the context window grows, attention degrades, and the model loses track of earlier insights. Conversely, independent "best-of-N" runs fail to accumulate knowledge, forcing each agent to rediscover insights from scratch.

2. Methodology: The Engram Architecture

The authors propose Engram, an agentic researcher architecture designed to decouple long-horizon exploration from the constraints of a single context window. Engram organizes the search process into a sequence of agents that iteratively design, test, and analyze mechanisms.

Core Components

Sequential Agent Explorations: Instead of a single long-running agent, Engram deploys a sequence of agents. Each agent operates within a fresh context window, preventing context degradation.
The Research Digest (Persistent Knowledge): At the end of each run, the agent distills high-level modeling insights, failure diagnoses, and key findings into a compact, structured Research Digest. This serves as a persistent handoff mechanism.
The Archive (Raw Data): Agents store code snapshots, execution logs, and raw experimental results in a persistent Archive.
The Handoff Protocol:
1. A new agent starts with a fresh context.
2. It reads the Research Digest from the previous agent to understand the problem state, promising directions, and dead ends.
3. It fetches specific code or logs from the Archive on demand (via tool calls) rather than embedding them in the context.
4. It formulates a new hypothesis, implements it, runs experiments, and analyzes results.
5. It updates the Research Digest and Archive before handing off to the next agent.

Key Design Principles

Scientific Method Loop: Agents follow a strict Hypothesize → Implement → Experiment → Analyze cycle.
Struggle Protocol: If an agent encounters a temporary regression (e.g., a cost spike during optimization), it is instructed to debug and analyze why the failure occurred rather than immediately reverting to a simpler heuristic. This allows the system to persist through non-monotonic progress toward a better solution.
Decoupling: By separating the "memory" (Digest/Archive) from the "reasoning" (Context Window), Engram achieves high coherence without context limits.

3. Key Contributions

Enhanced Long-Horizon Coherence: Introduced a structured handoff mechanism (Research Digest) that allows cumulative progress across agents, ensuring modeling insights persist beyond the lifetime of any single agent.
Discovery of Novel System Heuristics: Engram discovered heuristics that surpass human state-of-the-art (SOTA) and other AI baselines in multiple domains.
Superior Performance Across Diverse Problems: Evaluated on nine systems problems (including ADRS benchmarks), Engram outperformed human SOTA in 8 out of 9 settings and exceeded or matched OpenEvolve in all categories.

4. Experimental Results

The authors evaluated Engram on three primary case studies and a broader set of ADRS benchmarks:

A. Multi-Cloud Multicast

Goal: Minimize data transfer cost across cloud regions under time constraints.
Results:
- Engram: Achieved a best cost of $622, surpassing the human SOTA (Cloudcast) of $626 and all evolutionary baselines (OpenEvolve, FunSearch, EoH).
- Insight: Engram successfully navigated a "non-monotonic" path where intermediate attempts increased cost (e.g., jumping from $772 to $1104) before refining the approach to a tractable Mixed Integer Linear Program (MILP) that yielded the final win. Evolutionary methods remained stuck in Steiner-tree heuristics ( $640–$ 696).
- Prompt Sensitivity: Engram performed well even with minimal prompts, whereas evolutionary methods required specific "Direction" prompts to perform reasonably.

B. LLM Inference Request Routing

Goal: Route requests among replicated LLM instances to minimize mean response time.
Results:
- Engram: Achieved a mean response time of 23.9s, outperforming the expert-designed heuristic, Glia (25.7s), and all evolutionary baselines.
- Insight: Engram discovered a complex policy combining Shortest-Prefill-First ordering, decode-sensitive queue caps, and memory-utilization admission guards, explicitly reasoning about memory pressure and decode-phase contention.

C. KV Cache Reuse in Databases (LLM-SQL)

Goal: Reorder rows and columns in a dataframe to maximize KV cache reuse for LLM inference.
Results: Engram achieved the highest mean best score (0.721) and converged faster than OpenEvolve and Glia. It discovered non-recursive methods that leveraged per-row column reordering effectively.

D. Ablation Studies

Single Agent vs. Multi-Agent: A single agent with full tools performed worst (avg cost $902) due to context exhaustion.
Digest Importance: Removing the Research Digest caused performance degradation, proving that structured knowledge transfer is more critical than raw code transfer.
Persistence: Engram's ability to tolerate temporary regressions (the "Struggle Protocol") was identified as a key differentiator from methods that prune non-improving candidates immediately.

5. Significance and Conclusion

The paper demonstrates that persistence (accumulating knowledge across runs) and coherence (maintaining a consistent reasoning thread) are both necessary for AI to solve complex system design problems.

Paradigm Shift: Engram moves beyond simple code mutation (evolutionary) or isolated reasoning (single-agent) to a collaborative, multi-agent research process.
Human-Level Performance: By allowing agents to "fail forward" and learn from intermediate regressions, Engram can discover algorithmic families (like MILP or Dynamic Programming) that heuristic-based evolution often misses.
Generalizability: The architecture is domain-agnostic, successfully optimizing problems ranging from network routing to database query planning.

In summary, Engram provides a robust framework for automating systems research by treating the design process as a cumulative scientific inquiry rather than a simple optimization loop, enabling AI agents to achieve and surpass human expert performance in complex heuristic design.