Technical Summary: OSL-MR – Observability-Safe Memory Retention

1. Problem Definition

Long-horizon language agents face a fundamental resource-allocation challenge: they accumulate observations, reasoning traces, and retrieved facts that exceed finite context windows. Existing memory systems typically rely on heuristic scoring (e.g., recency, relevance), retrieval optimization, or learned compression. However, these approaches largely treat memory retention as a local decision problem, failing to explicitly model long-term consequences such as missing future evidence, reacquisition costs, and the risk of stale information under realistic partial observability.

A critical barrier to solving this is observability. Signals required to evaluate the quality of a retention decision (e.g., gold evidence, answer correctness, semantic freshness) are often only available after the decision is made. Using these signals during deployment creates an unrealistic information advantage (oracle access). Prior work has not explicitly formulated retention as a constrained stochastic optimization problem that respects the separation between online-observable features and offline-available supervision (OAS).

2. Methodology: OSL-MR

The authors propose OSL-MR (Observability-Safe Learning for Memory Retention), a framework that formulates memory retention as a constrained sequential optimization problem and solves it via a data-driven approach that strictly adheres to observability constraints.

2.1 Constrained Optimization Formulation

The problem is defined as maximizing expected cumulative reward over a horizon $T$ subject to a strict storage budget $B_t$. The reward function $r_t$ explicitly models:

• Evidence Utility: Rewards for hitting ($T^{hit}$) and achieving full coverage ($F_t$) of required evidence.
• Costs: Penalties for storage ($\alpha_{store}$), missing evidence ($\alpha_{miss}$), reacquisition delays ($\alpha_{reacq}$), and using stale information ($\alpha_{stale}$).

This formulation highlights that retention is a multi-step, delayed-feedback problem. A myopic, single-step optimizer cannot account for how current retention choices affect future evidence availability or incur delayed penalties.

2.2 Observability Separation (OAS)

The framework enforces a strict separation between:

• Online-observable inputs: Query context, memory metadata, interaction history, and recency. These are available at decision time.
• Offline-available supervision (OAS): Gold evidence sets, answer text, and ground-truth freshness. These are used only for training and evaluation, never for inference.

This ensures that any learned policy is deployable in real-world systems without requiring oracle access to future information.

2.3 Two-Component Architecture

To address the intractability of exact optimization under partial observability, OSL-MR employs two complementary components:

1. Mixed-Score Heuristic (Cold-Start & Prior):

   • A lightweight, fully deployable heuristic that computes a utility score based on temporal, relevance, context, and risk features, minus a size penalty.
   • Role 1: Serves as a strong baseline and ensures system functionality during the "cold-start" phase before interaction data is available.
   • Role 2: Acts as a structured inductive prior. Its score is concatenated with online features as an input to the learning model, guiding the learner without introducing new supervision signals.

2. Evidence Learner:

   • Trained offline on interaction logs collected by the Mixed-Score heuristic.
   • Supervision: Uses gold evidence membership labels ($y^{evid}_{t,i} = 1[m_{t,i} \in E_t]$) derived from realized outcomes.
   • Training: Optimizes a weighted binary cross-entropy loss to predict whether a memory is part of the required evidence set for a given query.
   • Deployment: The trained model is frozen and deployed, relying solely on online features (concatenated with the Mixed-Score prior) for inference.

2.4 Staged Deployment

The system operates in a practical staged manner:

1. Phase 1: Mixed-Score runs alone, logging all interactions.
2. Phase 2: Once sufficient data is collected, the Evidence Learner is trained offline using gold labels.
3. Phase 3: The frozen learner replaces the heuristic, maintaining strict observability constraints.

3. Key Contributions

1. Formalization: The first work to formalize memory retention as a constrained stochastic optimization problem for long-horizon agents, explicitly modeling evidence utility, delayed costs (miss, reacquisition, staleness), and partial observability.
2. OSL-MR Framework: A novel learning framework that enforces a strict online/OAS separation, ensuring deployability. It integrates a Mixed-Score heuristic (as a baseline and prior) with an evidence learner trained from direct evidence supervision.
3. Empirical Validation: Demonstrated on LoCoMo and LongMemEval benchmarks, showing consistent outperformance over recency-based methods, Generative Agents-style scoring, and behavior-cloning baselines, particularly under tight memory budgets.

4. Experimental Results

Experiments were conducted on two public long-horizon benchmarks with varying memory budgets (tight to loose).

• Performance: OSL-MR consistently achieved higher Evidence F1 and Reward than all baselines (Recency, Generative Agents, Mixed-Score, and Behavior Cloning).
  • Example (LoCoMo, Budget 128): OSL-MR achieved an F1 of 0.302 and Reward of 305.16, significantly outperforming Mixed-Score (F1 0.069, Reward 132.48) and Generative Agents variants (F1 < 0.01).
• Budget Efficiency: Unlike heuristic baselines that saturate the budget with low-utility items (high occupancy), OSL-MR maintains lower occupancy while achieving higher precision. It selectively discards marginal memories, prioritizing evidence quality over quantity.
• Supervision Quality: Direct evidence supervision (OSL-MR) significantly outperformed Behavior Cloning (BC-learner), which imitated a non-deployable "teacher" policy. This confirms that learning directly from gold evidence labels is more effective than imitating improved heuristics that rely on oracle information.
• Ablation (Mixed-Score Prior): Removing the Mixed-Score prior from the input features resulted in a drop in Precision (e.g., -20% on LongMemEval at budget 256) while Recall remained stable. This demonstrates the prior's role in improving selection efficiency without sacrificing coverage.
• Robustness: Sensitivity analysis across various cost configurations (varying miss, reacquisition, and staleness penalties) showed that OSL-MR remains the top-performing non-oracle method, indicating robustness to different cost trade-offs.

5. Significance and Claims

The paper claims that OSL-MR provides a unified perspective connecting heuristic scoring, optimization, and learning under a consistent observability constraint.

• Theoretical Significance: It addresses the gap in prior work where retention was treated as a local decision or a single-step optimization. By framing it as a multi-step sequential decision problem, OSL-MR captures the essential long-horizon nature of memory retention.
• Practical Significance: The strict observability separation guarantees that the learned policies are deployable in real-world interactive agents without requiring future information.
• Limitations & Future Work: The authors note that the Mixed-Score prior is currently manually designed and that freshness estimation relies on type-based decay. Future work aims to automate the learning of prior weights and develop more principled freshness estimation.

In summary, OSL-MR demonstrates that by formulating memory retention as a constrained optimization problem and strictly separating online inference from offline supervision, agents can learn to retain the "right" memories for long-horizon tasks, outperforming both static heuristics and oracle-dependent learning methods.