🤖 AI

Neuro-Symbolic Meta-Policies for Temporal Knowledge-Graph Memory under Partial Observability

This paper introduces a neuro-symbolic meta-policy that leverages temporal knowledge graphs and RDF-based memory representations to dynamically select symbolic memory heuristics for managing retention, retrieval, and forgetting in partially observable environments, achieving superior long-term performance and step-level traceability.

Original authors: Taewoon Kim, Vincent François-Lavet, Michael Cochez

Published 2026-07-22

📖 7 min read🧠 Deep dive

CC BY 4.0

Original authors: Taewoon Kim, Vincent François-Lavet, Michael Cochez

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to solve a giant, shifting maze, but you can only see the tiny room you are standing in right now. You can't see the whole map, and you can't remember every single turn you've ever taken because your brain has a limit on how much it can hold at once. This is the challenge of "partially observable" environments in the world of Artificial Intelligence. To navigate these mazes, AI agents need a way to decide what to keep in their memory, what to throw away, and how to find the right piece of information when they need it. If they remember too much, they get overwhelmed; if they forget too much, they get lost. The big question for scientists is: How do we build an AI that is smart enough to manage its own memory without becoming a black box that no one can understand?

This is where the paper "Neuro-Symbolic Meta-Policies for Temporal Knowledge-Graph Memory under Partial Observability" comes in. The researchers, Taewoon Kim, Vincent François-Lavet, and Michael Cochez, tackled this problem by creating a hybrid system that acts like a very organized librarian who also has a super-intuitive gut feeling. They built an AI that doesn't just guess; it uses a structured "knowledge graph" (think of it as a giant, interconnected web of facts) to store its memories. But here's the twist: instead of letting the AI randomly decide what to do, they taught it a "meta-policy." This is like giving the AI a menu of proven strategies—like "remember the newest thing," "remember the thing you used most often," or "forget the oldest thing"—and letting the AI learn which strategy to pick at any given moment. The result is a system that is both adaptable (it learns what works) and transparent (we can see exactly which rule it chose and why).

The Problem: The AI with a Bad Memory

Imagine you are playing a video game where you have to find a hidden key to open a door. The game world is huge, but you can only see the room you are currently in. As you walk around, you pick up clues: "The key was in the kitchen," or "The door is to the north." But your backpack (your memory) can only hold 512 items. If you pick up a 513th item, you have to drop something else.

If you drop the wrong thing, you might forget where the key is and fail the game. If you keep everything, your backpack gets too heavy, and you can't move. In the past, scientists tried two main ways to solve this. One way was to use strict, pre-written rules (like "always drop the oldest item"). This is reliable but rigid; it doesn't adapt if the game changes. The other way was to use a "black box" neural network, where the AI learns to remember everything on its own. This is flexible, but it's like a magic trick: you know the answer is right, but you have no idea how the AI decided to keep that specific fact and drop that other one. It's hard to trust or fix if it makes a mistake.

The Solution: The Smart Librarian with a Menu

The authors introduced a new approach called a "neuro-symbolic meta-policy." Let's break that down with a simple story.

Imagine your AI agent is a librarian in a library that is constantly being rearranged. The library is the "Temporal Knowledge Graph." Every book (fact) in the library has a label with three pieces of information: when it was added, when it was last read, and how many times it was borrowed. This is the "symbolic" part—the facts are clear, labeled, and organized.

Now, the librarian needs to decide what to do every second. Instead of guessing, the librarian has a menu of three types of decisions to make:

Question Answering: "Where is the red key?" The librarian can choose to look for the newest note about the key, the most recently used note, or the most frequently used note.
Exploration: "Where should I go next?" The librarian can choose to explore based on the newest map, the most visited map, or the most used map.
Forgetting: "My shelf is full! What do I throw away?" The librarian can choose to throw away the oldest item, the least recently used item, or the least frequently used item.

The "neuro" part of the system is the librarian's brain. It looks at the current situation (the room you are in, the question you are asking) and uses a special type of brain network (a Graph Neural Network) to score every option on the menu. It learns, through trial and error, which menu item to pick to get the best score.

The Experiment: The RoomKG Game

To test this, the researchers used a benchmark called "RoomKG." It's a grid world with 49 rooms, filled with objects like beds, lamps, and people. The AI has to navigate this world, answer questions like "Where is the lamp?", and move around, all while keeping its memory under the limit of 512 items.

They tested their new "Smart Librarian" against two other types of players:

The Rule-Follower: An AI that just uses fixed rules (like "always forget the oldest item").
The Black Box: An AI that tries to learn everything from scratch without any clear rules.

The Results: Adaptability Meets Clarity

The results were quite clear. The "Black Box" AI struggled significantly, scoring much lower than the others. It seems that trying to learn the entire complex action space (choosing a room and a direction out of 245 possibilities) all at once was too hard for it to handle with limited memory.

The "Rule-Follower" did well, proving that the basic structure of the memory system was sound. However, the "Smart Librarian" (the neuro-symbolic meta-policy) performed the best. Specifically, the version that used a "qualifier-aware" encoder (StarE-GNN) achieved the highest scores.

What makes this special is that the AI didn't just get lucky. Because the system is "neuro-symbolic," we can actually watch the librarian's thought process. The researchers could see that early in the game, the AI preferred to look at the "most recently used" facts because those were likely still relevant. But as the game went on and the world changed, the AI learned to switch to looking at the "newest" facts. It dynamically changed its strategy based on what was happening, something the rigid rule-followers couldn't do.

Why This Matters

The most exciting part of this paper isn't just that the AI got a higher score. It's that it did so while remaining completely transparent. In many AI systems, if you ask, "Why did you forget that fact?" the answer is usually, "Because the math said so." In this system, the answer is, "I chose the 'Least Frequently Used' rule because I calculated that fact was rarely needed."

The authors suggest that this approach offers a sweet spot: the adaptability of a learning AI combined with the clarity of a rule-based system. They showed that by teaching an AI to select the right "tool" from a toolbox of known strategies, rather than trying to invent a new tool from scratch every time, you can get better performance without losing the ability to understand how the AI thinks.

While this was tested in a specific game environment with a memory limit of 512, the idea is that this "meta-policy" approach could be applied to other situations where an AI needs to manage a lot of information over time. It proves that you don't have to sacrifice understanding for performance; you can have an AI that is both smart and explainable.

Problem Setting

The paper addresses the challenge of Partially Observable Reinforcement Learning (POMDP) where an agent must manage a bounded long-term memory to make decisions over long horizons. The specific environment is RoomKG, a grid-world benchmark where the agent's hidden state and observations are represented as Resource Description Framework (RDF) graphs.

In RoomKG, the agent receives only local observations (the current room and immediate adjacencies) as RDF subgraphs, while the full environment state (including object locations in other rooms and dynamic changes) remains hidden. The agent must answer queries about object locations and navigate the environment. Crucially, the agent operates under a strict long-term memory capacity constraint (set to 512 facts in this study). The core difficulty lies in coordinating three competing tasks:

Retrieval: Deciding which stored facts to recall to answer queries.
Exploration: Deciding which parts of the remembered map to investigate.
Forgetting: Deciding which facts to evict when memory capacity is reached.

Traditional deep RL approaches often use latent memory (e.g., LSTMs), which are difficult to inspect or trace. Conversely, fixed symbolic heuristics are interpretable but inflexible across different contexts. The paper seeks a solution that retains the inspectability and explicit reasoning of symbolic systems while achieving the adaptivity of learned policies.

Methodology

The authors propose a Neuro-Symbolic Meta-Policy architecture. Instead of learning a direct policy to output environment actions (e.g., "move north" or "answer room 5"), the agent learns a meta-policy that selects among named symbolic heuristics at each decision point.

1. Memory Representation:
The agent maintains a temporal knowledge graph ( $M_t$ ) composed of short-term and long-term memories. Facts are stored as RDF triples augmented with temporal annotations (qualifiers) such as time_added, last_accessed, and num_recalled. This allows the system to treat memory management as a selection problem over explicit metadata.

2. Symbolic Action Space:
The meta-policy selects a joint symbolic action $h_t = (h_{forget}, h_{qa}, h_{explore})$ from a finite product space of 27 combinations ( $3 \times 3 \times 3$ ):

QA Heuristics: Most-Recently-Added (MRA), Most-Recently-Used (MRU), Most-Frequently-Used (MFU).
Exploration Heuristics: Same ranking priorities (MRA/MRU/MFU) applied to planning.
Forgetting Heuristics: First-In-First-Out (FIFO), Least-Recently-Used (LRU), Least-Frequently-Used (LFU).

3. Neural Architecture:
The system uses a Graph Neural Network (GNN) encoder to process the memory graph and feed value heads that score the symbolic heuristics. Three encoder variants are compared:

GCN: Topology-only encoding.
R-GCN: Relation-aware encoding.
StarE-GNN: Qualifier-aware encoding (incorporating the temporal annotations directly into message passing).

The architecture includes three independent value heads (or a single combinatorial head in ablation studies) that output Q-values for the respective heuristic sets. The selected heuristics are then executed deterministically by a symbolic controller ( $\Phi$ ) to update the memory state and choose the environment action.

4. Training:
The system is trained using off-policy Q-learning with temporal-difference (TD) updates.

QA Head: Trained as a contextual bandit (immediate reward) since correctness is determined instantly.
Explore/Forget Heads: Trained with TD learning to assign credit to decisions that affect future QA success.

Key Contributions

Neuro-Symbolic Meta-Policy Formulation: The paper introduces a framework where adaptation is achieved by learning to select among explicit, named symbolic memory heuristics, rather than replacing symbolic control with opaque latent actions.
Qualifier-Aware Temporal Memory: The design integrates RDF-based memory with temporal annotations, using qualifier-aware graph encoders (specifically StarE-GNN) to align the neural representation with the symbolic decision problem.
Traceability and Performance: The approach demonstrates that it is possible to achieve state-of-the-art performance in a bounded-memory setting while preserving step-level traceability of memory-management decisions.

Results

Experiments were conducted on the RoomKG benchmark with a long-term memory capacity of 512.

Performance: The StarE-GNN-based neuro-symbolic configuration achieved the best held-out test performance (47.28% QA success), outperforming both fixed symbolic baselines (which ranged from ~43% to ~46%) and end-to-end neural baselines (LSTM/Transformer, which performed poorly at ~9–18%).
Encoder Impact: Qualifier-aware encoders (StarE-GNN) and relation-aware encoders (R-GCN) significantly outperformed topology-only encoders (GCN), confirming that preserving relation and qualifier semantics is critical for the decision problem.
Modularity vs. Combinatorial: The modular design (three independent 3-way heads) outperformed a single 27-way combinatorial head, suggesting that independent supervision for each decision type (QA, explore, forget) is more effective.
Interpretability: Analysis of Q-value trajectories showed that the meta-policy adaptively switches between heuristics (e.g., shifting from MRU to MRA preference as the episode progresses) based on the memory context, providing a transparent view of the agent's reasoning.

Significance and Claims

The paper claims that adaptivity and inspectability need not be traded off. By learning a meta-policy over symbolic heuristics, the system improves upon strong fixed symbolic controls and opaque neural baselines while maintaining an explicit, RDF-grounded memory state.

The authors position this work as a direct grounding in the Semantic Web, utilizing RDF for representation, annotation-compatible graph semantics, and graph-based symbolic operations. They emphasize that the contribution is the meta-policy layer and its evaluation, not the underlying benchmark or memory substrate.

The scope is modestly defined: the results are specific to the RoomKG environment and a memory capacity of 512. The authors acknowledge limitations, including the reliance on a specific library of heuristics (performance is capped by the quality of available heuristics) and the current focus on deterministic dynamics rather than stochastic or non-stationary regimes. Future work is suggested to broaden environments, test stochastic robustness, and expand heuristic libraries while retaining explicit decision traces.