Technical Summary: ForecastCompass (FOCO)

Problem Statement

Agentic forecasting involves Large Language Model (LLM) agents making calibrated probabilistic predictions about future events using incomplete and time-limited evidence. While LLM agents excel at information seeking and reasoning, they face significant challenges in dynamic forecasting environments. Specifically, agents must reason from evolving evidence and produce calibrated probabilities before outcomes are resolved.

Existing memory mechanisms for LLM agents are primarily designed for question-answering or problem-solving tasks. These methods typically store past interactions, reflections, or factual associations. However, they are ill-suited for forecasting because they fail to explicitly represent reusable predictive factors or calibration knowledge. A generic memory might store that an app has "strong brand recognition," but it fails to distinguish whether this static signal is informative for short-term rank changes or if it should be treated as a misleading signal that warrants lower confidence. Forecasting requires memory that can abstract predictive factors, filter misleading signals, and calibrate uncertainty before outcomes are known, rather than simply retrieving past answers or event-level trajectories.

Methodology: ForecastCompass (FOCO)

The authors propose ForecastCompass (FOCO), an adaptive factor-based memory framework designed specifically for agentic forecasting. FOCO organizes forecasting experience into a hierarchical structure and updates it chronologically using retrospective analyses as learning signals.

1. Hierarchical Forecasting Memory Representation

FOCO organizes memory using a forecasting taxonomy consisting of categories and fine-grained subcategories. For each subcategory $s$, the system maintains a verbal memory state $M_s = (F_s, R_s)$ comprising two complementary components:

• Factor Memory ($F_s$): A set of entries describing reusable predictive dimensions. Each entry includes the factor name, specific evidence checks, common failure modes, and the factor's typical effect on forecast probabilities. This guides the agent in identifying relevant evidence and avoiding misleading patterns.
• Reasoning Memory ($R_s$): A collection of calibration-oriented principles. This component encodes how confidence should be adjusted under different evidence conditions, handling uncertainty, conflicting signals, and incomplete information. It guides probability estimation rather than just factor selection.

2. Memory-Augmented Inference

During inference, an unresolved question is classified into a specific subcategory. The agent retrieves the corresponding $F_s$ and $R_s$ to guide its process:

• Factor Memory directs evidence search and factor selection, emphasizing predictive signals and warning against misleading patterns.
• Reasoning Memory provides calibration principles for translating evidence strength into probabilistic forecasts, helping to manage overconfidence or underconfidence.

3. Iterative Memory Revision via Contrastive Learning

FOCO updates its memory chronologically using resolved forecasting tasks. The update process involves a diagnose–aggregate–revise procedure:

1. Retrospective Trajectory Generation: After a question is resolved, the agent generates a retrospective trajectory ($z^{retro}$) using post-resolution evidence. This serves as a "ground truth" learning signal.
2. Contrastive Diagnosis: The system contrasts the original forecasting trajectory ($z$) with the retrospective trajectory ($z^{retro}$) to identify discrepancies ($\Delta$).
   • $\Delta^F$: Factor-level discrepancies (e.g., missing factors, overused misleading signals).
   • $\Delta^R$: Reasoning-level discrepancies (e.g., miscalibration, incorrect probability updates).
3. Aggregation and Revision: The system aggregates these discrepancies across multiple events within a subcategory to filter out event-specific hindsight (noise) and identify recurring patterns. It then revises the verbal memory entries to incorporate these transferable predictive factors and calibration principles, explicitly avoiding the storage of event-specific outcomes or dates.

Key Contributions

1. Factor-Centric Memory Formulation: The paper introduces a memory architecture organized around subcategory-level predictive factors and calibration-oriented reasoning patterns, distinct from event-level trajectories or generic reflections.
2. Verbalized Factor-Memory Revision: The authors develop an iterative procedure that treats retrospective analyses as learning signals. This process diagnoses discrepancies between forecasts and outcomes, aggregates them to find recurring patterns, and revises memory into transferable abstractions while excluding event-specific hindsight.
3. Empirical Validation: The framework is evaluated on dynamic forecasting benchmarks (Prophet Arena and FutureX) using GPT-5-mini and Gemini-2.5-Flash.

Experimental Results

Experiments demonstrate that FOCO improves both probabilistic accuracy and calibration compared to various baselines:

• Performance: FOCO consistently achieves the lowest average Brier scores and Expected Calibration Error (ECE) across both datasets and backbone models. It outperforms non-memory baselines (BASE), static memory variants (FOCO Static), and general agent-memory baselines (MEM0, REFLEXION, A-MEM, GRAPHITI).
• Iterative Improvement: The full FOCO framework outperforms the static version, indicating that weekly memory revision effectively refines predictive factors and reasoning patterns as new outcomes become available.
• Transferability:
  • Across-Time: Memory constructed from earlier weeks improves forecasting performance in later weeks, demonstrating that the learned factors and calibration principles remain relevant as topics evolve.
  • Across-Model: Memory constructed with one model (e.g., Gemini-2.5-Flash) successfully improves the performance of a different backbone (e.g., GPT-5-mini), suggesting the captured knowledge is not overfit to a specific model architecture.
• Memory Quality: An LLM-as-a-Judge evaluation shows FOCO memory scores higher on generalizability, novelty, completeness, accuracy, and actionability compared to other memory methods, confirming it captures reusable forecasting knowledge rather than specific event details.

Significance and Claims

The paper claims that effective forecasting memory must go beyond storing past interactions or answers. Instead, it must abstract reusable predictive factors and calibration principles that guide agents in open-world settings where evidence is unstructured and evolving.

FOCO addresses the gap in existing agent-memory designs by explicitly separating predictive signals from reasoning patterns and using contrastive retrospective analysis to iteratively refine these abstractions. The results suggest that organizing memory around factor-level knowledge and calibration rules allows agents to accumulate transferable forecasting expertise over time, leading to more accurate and better-calibrated probabilistic predictions in dynamic environments. The work highlights that simply retrieving past agent experiences is insufficient; the memory structure itself must be tailored to the specific demands of probabilistic forecasting under uncertainty.