💻 computer science

Bridging Inference-Time Scaling and Episodic Memory with Action-Centric Graphs

The paper introduces GAMER, a novel framework that bridges inference-time scaling and episodic memory by modeling historical reasoning as an action-centric graph with a dual-stream Temporal Difference learning mechanism, thereby reducing computational costs and significantly improving agent decision-making efficiency and success rates.

Original authors: Xu Zheng, Chaohao Lin, Zhuomin Chen, Weijieying Ren, Haifeng Chen, Wei Cheng, Dongsheng Luo

Published 2026-07-31

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xu Zheng, Chaohao Lin, Zhuomin Chen, Weijieying Ren, Haifeng Chen, Wei Cheng, Dongsheng Luo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a super-smart robot to solve a maze. You tell it, "Think hard, try different paths, and don't give up!" The robot, being a Large Language Model (LLM), is great at thinking. It can generate a long list of ideas, like "Turn left, then right, then maybe climb the wall." This is called inference-time scaling: giving the robot more time and computing power to brainstorm before it moves. But here's the catch: every time the robot hits a dead end, it forgets everything. It starts the next attempt with a blank slate, re-trying the same wrong turns over and over again. It's like a student taking a test who erases their entire brain after every question, forcing them to re-learn the same mistakes from scratch.

To fix this, scientists have tried giving the robot a memory. Usually, this memory is just a giant notebook where the robot writes down everything it did. But reading through a massive notebook takes a lot of time and energy (computing power), and the robot still has to figure out how to use that information. The big question is: How do we give a robot a memory that actually helps it think faster and smarter, without making it slower or more expensive? This paper tackles that exact problem by building a new kind of memory that doesn't just store stories, but maps out the actual moves that work.

Meet GAMER (Graph-based Action-centric Memory with Episodic Reasoning). Think of GAMER as a "cheat sheet" for a video game, but instead of just listing levels, it draws a map of every single move you've ever made.

Most robots today are like explorers in a foggy forest. When they need to find a treasure, they wander around, try a path, hit a tree, turn back, and try again. If they fail, they forget the tree was there. Next time, they wander into the same tree. GAMER changes the game by turning the robot's past into a living map.

Instead of writing down long stories like "I walked left, saw a dog, then walked right," GAMER builds a Graph. Imagine a subway map where every station is a specific action the robot can take (like "pick up the key" or "open the door"). The lines connecting the stations show which actions usually follow others. If the robot tries to "jump over a wall" and fails, that station on the map gets a big red "X" and a warning sign. If it tries to "climb the ladder" and succeeds, that station gets a glowing green star.

The magic happens with a special learning trick called Dual-Stream TD Learning. This is like having two coaches giving the robot advice at the same time:

The "Go" Coach: This coach looks at all the times the robot succeeded. They point to the glowing green stars and say, "Hey! If you are at this spot, try this move next. It usually leads to a win!"
The "Stop" Coach: This coach looks at all the times the robot crashed. They point to the red "X" stations and say, "Absolutely not! If you see this move, run the other way. It leads to a dead end."

By using these two coaches, GAMER doesn't just tell the robot what to do; it actively stops the robot from wasting time on bad ideas. It's like having a GPS that not only shows you the fastest route but also instantly blocks off roads you know are under construction.

The researchers tested this on several challenging tasks, like organizing a messy room (AlfWorld) or solving science experiments (ScienceWorld). They found that GAMER was a game-changer. Compared to the standard "forgetful" robot, GAMER improved the success rate by 20.81% and the progress rate by 6.17%. That means the robot finished more tasks and got further in the ones it didn't finish.

Even cooler, GAMER is efficient. Because it uses this smart map instead of reading a giant notebook, it saves a lot of "tokens" (the digital currency of AI thinking). In fact, compared to another smart memory system called A-Mem, GAMER used about 50% fewer tokens on average. It's like solving a puzzle with a clear diagram instead of trying to remember every piece of the box lid.

The paper shows that by turning past experiences into a structured map of "good moves" and "bad moves," robots can learn much faster. They don't have to re-invent the wheel every time they face a problem. However, the authors note that this system needs a little bit of practice time first. The robot needs to see about 32 successful attempts (a "warm-up") to build a good map. If it doesn't have enough practice data, the map might be too empty to be helpful. But once that map is built, the robot becomes a much more efficient and successful explorer, navigating complex problems with a clear head and a reliable guide.

Technical Summary: GAMER (Graph-based Action-centric Memory with Episodic Reasoning)

Problem Statement

Recent advancements in inference-time scaling (e.g., Chain-of-Thought, Tree-of-Thoughts) have enhanced the reasoning capabilities of Large Language Models (LLMs) by allocating additional test-time compute to explore solution spaces. However, when applied to LLM-driven agents operating in persistent or recurring environments, these methods suffer from critical inefficiencies:

Stateless Operation: Current scaling methods operate "from scratch" for every new problem, ignoring historical search trajectories and prior experiences. This leads to redundant exploration of the same massive search space.
Computational Overhead: Existing memory mechanisms (e.g., A-MEM, G-MEM) often rely heavily on the LLM's internal reasoning capabilities to summarize, retrieve, and inject context. This results in prohibitive token usage and high latency, as the agent must reconstruct reasoning trees and evaluate branches ab initio even when relevant facts are retrieved.
Lack of Search Guidance: Traditional memory systems act as passive retrieval modules for declarative facts. They do not actively prune the search space or guide decision-making, leaving the dominant cost of redundant reasoning unaddressed.

Methodology: GAMER

The authors propose GAMER (Graph-based Action-centric Memory with Episodic Reasoning), a framework that decouples memory from the LLM's reasoning process. Instead of storing raw text logs, GAMER models historical reasoning as a dynamic Action-Centric Graph and uses Temporal Difference (TD) learning to estimate action values.

1. Action-Centric Graph Construction

Structure: The framework aggregates historical interaction trajectories into a directed graph $G = (V, E)$ $G = (V, E)$ .
- Nodes ( $V$ ): Represent distinct, unique actions executed by the agent.
- Edges ( $E$ ): Represent temporal transitions observed in history (i.e., action $a_j$ immediately following $a_i$ ).
Advantage: This condenses linear experiences into a global map of the reasoning space, capturing sequential dependencies and decision points without relying on the LLM to parse unstructured text for memory extraction.

2. Dual-Stream Temporal Difference (TD) Learning

To navigate the graph effectively, GAMER employs a dual-stream value estimation mechanism to capture both potential rewards and risks, addressing the limitation of single-scalar value functions that might neutralize critical risk signals.

Positive Value ( $Q^+$ ): Estimates the expected success rate of an action based on standard environmental rewards. It identifies high-value trajectories.
Negative Value ( $Q^-$ ): Estimates the risk of failure. It uses a Thresholded Reward Redefinition, assigning a negative reward ($-1$) if an action leads to sub-threshold performance or failure, and $0$ otherwise.
Update Mechanism: Both streams are updated iteratively using TD learning:
$Q(v_t) \leftarrow Q(v_t) + \alpha [r_t + \gamma \cdot \text{Target}(v_{t+1}) - Q(v_t)]$
This allows the graph to evolve into a nuanced map where high-value paths are reinforced and high-risk "dead ends" are explicitly flagged.

3. Memory-Guided Inference Scaling

During the inference phase for a new task, the agent queries the graph to guide the LLM via In-Context Learning (ICL) without modifying the underlying decoding process:

Reference Trajectory: Retrieves the historically optimal trajectory ( $\tau^*$ ) as a few-shot example.
Dual-Stream Action Injection:
- Suggestions ( $A_{suggest}$ ): Injects the Top-K actions with the highest $Q^+$ values as positive guidance.
- Avoidance ( $A_{avoid}$ ): Injects the Top-K actions with the most negative $Q^-$ values as warnings.
Prompt Integration: These signals are converted into natural language instructions within the prompt, steering the LLM's probability distribution toward high-value regions and suppressing error-prone branches.

Key Contributions

Paradigm Shift: Introduces a memory-augmented inference framework that constructs a task-specific Action-Centric Graph, shifting from retrieving static text to querying dynamic action values.
Bi-Directional Learning: Proposes a Dual-Stream TD Learning mechanism that allows agents to simultaneously learn high-value heuristics from successes and strict constraints from failures, ensuring continuous refinement of the search policy.
Efficiency and Performance: Validates the framework across multiple benchmarks, demonstrating that it outperforms strong baselines while significantly reducing token consumption compared to memory mechanisms that rely on LLM reasoning for extraction.

Experimental Results

The authors evaluated GAMER on four benchmarks: AlfWorld, ScienceWorld, PDDL, and Tool-Query, using four LLM backbones (Qwen2.5, Llama3.1, Gemma3, GPT-4o-mini).

Performance Gains: Compared to vanilla baselines (Best-of-N), GAMER improved the Success Rate by 20.81% and the Progress Rate by 6.17% on average.
Comparison to SOTA: GAMER consistently outperformed other memory-augmented and self-evolving methods (e.g., A-Mem, ReasoningBank, Voyager). For instance, on the AlfWorld benchmark, it achieved a 53.17% improvement over the stateless vanilla baseline.
Token Efficiency: While GAMER incurred slightly higher token costs than the vanilla baseline due to graph queries, it was significantly more efficient than other memory-based methods. Specifically, it saved approximately 50% of tokens compared to A-Mem and avoided the massive token overhead of ReasoningBank (which relies on iterative self-refinement).
Scaling Behavior: The method demonstrated consistent performance gains as the inference budget ( $N$ ) increased, maintaining the steepest performance slope compared to competitors, indicating effective utilization of additional compute.

Significance and Claims

The paper claims that GAMER bridges the gap between inference-time scaling and episodic memory by transforming memory from a passive storage of logs into an active, computational asset.

Decoupling: By decoupling memory from the LLM's reasoning ability, GAMER reduces the computational cost of memory management, making it feasible to leverage long-term history without prohibitive token usage.
Search Optimization: The framework acts as a "soft barrier" against known failure modes and a "guide" for successful paths, effectively pruning the search space before the LLM generates redundant reasoning steps.
Practicality: The authors assert that GAMER offers a more practical solution for real-world agent deployment, achieving stronger performance with lower token costs and better stability across different model scales compared to methods heavily dependent on the intrinsic reasoning capabilities of the LLM.

Limitations: The authors acknowledge that the method depends on sufficient warm-up trajectories to initialize the graph. If the initial knowledge set is too small, the agent may converge to suboptimal behaviors or become trapped in erroneous patterns, potentially degrading performance below the vanilla baseline.