Technical Summary: PM-Bench: Evaluating Prospective Memory in LLM Agents

Problem Statement

The paper identifies a critical gap in the evaluation of Large Language Model (LLM) agents: prospective memory. While existing benchmarks focus on long-horizon planning, tool use, and retrospective recall (retrieving past information), they fail to isolate the ability to execute an intention at a specific future cue or state while other activities are ongoing. The authors argue that agent failures often stem not from a lack of knowledge, but from the inability to act on that knowledge at the correct moment. This capability is distinct from simple recall because it requires maintaining intentions under distraction, monitoring latent environment changes, and calibrating the timing of actions without external reminders.

Methodology

PM-Bench Construction

The authors introduce PM-Bench, a text-based benchmark inspired by the Virtual Week paradigm from cognitive science. The benchmark simulates a seven-day week with 80 time-anchored steps.

• Task Structure: Agents must navigate a narrative while managing deferred intentions. Tasks are categorized as:
  • Event-based: Triggered by specific cues in the narrative or hidden state channels.
  • Time-based: Triggered when the virtual clock reaches a specific time.
  • Cross-day: Announced on one day but executable only on a future day.
  • Dynamic Updates: Tasks can be canceled, rescheduled, or overridden, requiring the agent to update its internal state.
• Interaction Protocol: The benchmark operates on a query-then-act protocol. At each step, the agent receives a narrative scene and three ongoing activity choices. To execute a prospective task, the agent must select from a menu of anonymous action handles. Crucially, many cues are hidden in state channels (e.g., email, clock, appointment portal) that the agent must proactively query.
• Validation: Scenarios are generated via a seeded synthetic pipeline and validated for solvability, ensuring that every task has a valid path to completion if the agent monitors the correct channels and acts at the right time.

Experimental Setup

The authors evaluated 8 state-of-the-art LLMs (including GPT-5.4, GPT-5.3-Codex, Llama-3.3, Mistral, and Qwen3 variants) across 8 different agent configurations:

1. Single-agent baseline: No external memory or scaffolding.
2. Todo-ledger: An in-context ledger tracking pending intentions and statuses.
3. Heartbeat variants: Agents prompted to periodically check hidden channels (optional, auto-30m, auto-60m).
4. Hierarchical multi-agent: A coordinator managing three specialist subagents (event watcher, status watcher, update watcher) who propose queries and tasks.
5. Voting ablations: Replay-based majority and unanimous voting over the hierarchical traces.

Evaluation Metrics

The primary metric is Set-F1, calculated over the entire trajectory. This metric balances:

• True Positives (TP): Correctly executing a due task.
• False Positives (FP): Executing a task that is not yet due (over-acting/spamming).
• False Negatives (FN): Failing to execute a due task (misses).
  The authors emphasize Set-F1 because high recall can be achieved by "spamming" actions, while high precision can be achieved by never acting; a robust agent must balance both.

Key Results

1. No Universal Best Strategy

No single agent configuration dominates across all models or metrics.

• The optional-heartbeat single agent achieved the highest overall Set-F1 (65.1%).
• The todo-ledger variant achieved the second-highest Set-F1 (62.8%) and the lowest false-positive count, indicating superior precision.
• Hierarchical multi-agent setups, despite issuing significantly more queries (1,661 total vs. ~100 for single agents), performed poorly on Set-F1 (45.2%), suggesting that increased monitoring does not automatically translate to better action calibration.

2. The Precision-Recall Tradeoff

The results reveal a fundamental tension in prospective memory:

• Over-acting: Methods that recover more due tasks (e.g., auto-heartbeat 30m) often increase false positives, leading to lower Set-F1 scores.
• Under-acting: Conservative methods preserve precision but miss valid opportunities.
• Model Dependence: The optimal scaffold varies by model backbone. For instance, GPT-5.4 performed best with optional heartbeat, while Qwen3-8B showed a significant jump with optional heartbeat (71.9%) compared to the baseline (42.0%), whereas other models showed different preferences.

3. Proactive Monitoring Remains a Bottleneck

• Hidden Channels: Agents struggle significantly with tasks requiring hidden-state monitoring. While clock-monitoring hit rates reached ~60% for top setups, hit rates for other hidden channels (e.g., email, shipment status) remained near 0–16%.
• Cross-Day and Updates: Long-horizon tasks (cross-day) and dynamic updates (reschedules/cancellations) remain largely unsolved. The best cross-day hit rate was only 50.0%, and no setup exceeded 50% on update-sensitive execution.
• Multi-Agent Limitations: The hierarchical setup failed to improve performance on narrative-cued tasks (dropping to 45.2% hit rate) despite high query volume, indicating that the bottleneck is not just acquiring evidence but converting it into correct action sets.

Significance and Claims

The paper claims that prospective memory should be treated as a distinct evaluation axis for LLM agents, separate from context window size or generic memory retrieval capabilities.

• Diagnostic Value: PM-Bench serves as a controlled testbed to diagnose specific failure modes, such as the inability to maintain deferred intentions across days or the failure to distinguish between "active" and "due" tasks.
• Calibration Challenge: The core finding is that prospective behavior is hard to calibrate. Current agents either forget to act or act too aggressively.
• Future Directions: The benchmark provides a foundation for developing training or inference-time interventions (e.g., better scaffolding, memory management) that support reliable, time-sensitive agent behavior.

The authors explicitly caution against interpreting high benchmark scores as evidence of readiness for safety-critical real-world deployment (e.g., healthcare or aviation), noting that prospective memory failures in those domains have severe consequences. PM-Bench is presented as a diagnostic tool to measure and mitigate these specific failure modes.