Technical Summary: Short-Term-to-Long-Term Memory Transfer for Knowledge Graphs under Partial Observability

Problem Formulation

This paper addresses the challenge of short-term-to-long-term memory transfer in Reinforcement Learning (RL) under Partially Observable Markov Decision Processes (POMDPs). Specifically, it focuses on environments where observations are explicit symbolic triples (Resource Description Framework, or RDF) rather than raw sensory data.

In the RoomKG benchmark, the agent operates in a partially observable world where the full state is hidden. At each step, the agent receives a local induced subgraph of the world (a set of observed triples). The core problem is deciding which of these observed facts should be transferred to a capacity-limited long-term memory and which should be dropped. This is framed as a neuro-symbolic decision problem: for every observed triple in the short-term buffer, the agent must make an explicit binary "keep" or "drop" decision before insertion into long-term storage.

A primary technical challenge is variable cardinality: the number of observed facts ($n_t$) changes at every time step, meaning the dimensionality of the decision space is not fixed. Standard RL action spaces are typically fixed-size, making direct application difficult. Furthermore, the paper isolates this transfer mechanism by keeping other components (question answering, exploration, and memory eviction) fixed to ensure performance differences are attributable solely to the transfer policy.

Methodology

The authors propose a per-item Q-learning approach designed to handle variable-sized short-term memory sets.

1. Per-Item Q-Value Parameterization

Instead of treating the set of short-term items as a single combinatorial action (which would result in $2^{n_t}$ actions), the method uses a shared-parameter function $Q_\theta$.

• Input: The current symbolic memory state $M_t$ is converted into a graph $G_t$ containing entities, relations, and temporal annotations.
• Encoding: A Graph Neural Network (GNN) encoder processes $G_t$ to produce node embeddings. For each short-term item $m_{t,i} = (u, r, v, \xi)$, the representation $z_{t,i}$ is formed by concatenating the embeddings of the head ($u$) and tail ($v$) nodes. This allows the representation to implicitly capture context from neighboring triples via the GNN's message passing.
• Output: A small Multi-Layer Perceptron (MLP) head maps $z_{t,i}$ to a vector of Q-values for the two actions: $\{0, 1\}$ (drop or keep).
• Architecture Variants: The study evaluates different GNN backbones (GCN, R-GCN, StarE-GNN) and policy modes (Local vs. Global, Short-Term-only vs. Full-context).

2. Temporal-Difference (TD) Learning with Variable-Cardinality Matching

Standard off-policy TD learning assumes fixed-dimensional transitions. To handle the varying number of items ($n_t$) across steps, the authors introduce a stochastic matching strategy:

• Transitions are stored as $(M_t, a_t, r_t, M_{t+1}, d_t)$.
• During training, a mini-batch is sampled. For a pair of consecutive steps in a transition, the algorithm matches items index-wise up to the minimum length $\ell_b = \min(n_b, n'_{b})$.
• The TD target $y_{b,j}$ is computed for each matched index $j$ using the max Q-value of the next step.
• The loss is the mean squared TD error over these matched indices, normalized by $\ell_t$. This allows the model to learn from dense supervision without requiring a fixed action dimension or imposing artificial sequence semantics on the set-based memory.

3. Training Protocol

The system is trained using standard off-policy DQN techniques (Mnih et al., 2015) with $\epsilon$-greedy action selection. The reward signal comes from the downstream task (Question Answering correctness), meaning the transfer policy learns via delayed credit assignment to maximize future QA performance.

Key Contributions

1. Formalization of Transfer as an RL Problem: The paper formalizes short-term-to-long-term memory transfer as a neuro-symbolic RL problem where each observed fact receives an explicit keep/drop decision. This isolates the transfer mechanism from other policy components.
2. Per-Item Q-Learning for Variable Cardinality: The authors introduce a practical value-based training recipe for variable-cardinality memory decisions. By using shared parameters and stochastic index-wise matching for TD updates, they enable learning over sets of changing size without latent memory compression.
3. Empirical Validation on RoomKG: The study provides controlled evidence on the RoomKG benchmark (Long-term capacity = 128). It demonstrates that learned transfer decisions outperform:
   • Symbolic baselines with temporal annotations (e.g., Novel-Only, Random-Transfer).
   • History-based neural baselines (LSTM, Transformer).
   • Ablations of the proposed method (e.g., Global policies or full-context inputs performed worse than local short-term-only policies).

Results

On the RoomKG benchmark with a long-term memory capacity of 128:

• Performance: The best configuration (GCN encoder + Local-STM policy) achieved a test QA score of 38.92 ± 3.09. This significantly outperformed the strongest symbolic baseline (Novel-Only: 31.96) and neural history-based baselines (Transformer: 11.80; LSTM: 7.60).
• Ablation Insights:
  • Local vs. Global: Per-item local decisions (Local-STM) vastly outperformed pooled global decisions (Global-STM), suggesting that fact-specific context is crucial.
  • Input Scope: Using only short-term memory as input (STM) was more effective than using the full memory context (Full), indicating that the transfer decision relies heavily on immediate observation relevance rather than global state.
  • Encoder Choice: Standard GCNs outperformed more complex relational encoders (R-GCN, StarE-GNN), suggesting that for this specific transfer task, simple structural encoding is sufficient.
• Behavioral Analysis:
  • The policy learned to preserve navigation-critical facts (e.g., the agent's current location) with near-perfect retention (98/100 times).
  • It retained query-relevant object locations (58/60 times) while discarding many directional map links (e.g., north/south/east/west edges), treating them as lower-value or noisy under capacity constraints.
  • The overall keep rate was approximately 40%, demonstrating adaptive filtering rather than fixed heuristics.

Significance and Claims

The paper positions itself as a focused mechanism-validation study rather than a broad benchmark sweep. Its primary significance lies in demonstrating that:

1. Explicit, Interpretable Decisions: Unlike latent-memory approaches (e.g., DRQN, DMN) where state updates are opaque, this method produces explicit, inspectable keep/drop decisions for symbolic triples.
2. Effectiveness under Constraints: Learned transfer decisions can effectively manage memory pressure in POMDPs, outperforming both fixed symbolic heuristics and end-to-end sequence models when the memory capacity is constrained.
3. Practicality: The proposed per-item Q-learning with stochastic matching provides a viable training recipe for variable-cardinality decision problems in RL.

The authors modestly acknowledge limitations: the evaluation is restricted to the RoomKG environment and a specific memory capacity; non-transfer components were fixed to isolate the variable; and the TD matching strategy is a practical heuristic rather than a theoretically complete solution for set-based alignment. Future work is suggested to explore broader environments and joint end-to-end training while preserving fact-level interpretability.