Technical Summary: Freshness and the Limits of Heuristic Trend Detection in Temporal RAG

Problem Statement

Retrieval-Augmented Generation (RAG) systems typically rank evidence based solely on semantic similarity within an embedding space, ignoring temporal dynamics. This atemporal approach fails in domains where recency, historical validity, or the evolution of topics is critical (e.g., cybersecurity, sports analytics, news). Without a temporal component, RAG systems risk surfacing stale information, missing critical updates, or providing incorrect "as-of" statements. While vector embeddings excel at semantic proximity, they cannot distinguish between a novel update and redundant, outdated information.

Methodology

The authors propose a lightweight, model-agnostic temporal memory layer that integrates directly into existing RAG pipelines without retraining base models. The system operates through four distinct stages:

1. Data Normalization: Disparate input streams (e.g., JSONL logs, CSVs) are mapped to a unified schema containing a UTC timestamp, a compact text representation, and optional domain keys.
2. Embedding: Text representations are encoded into 384-dimensional vectors using a lightweight Sentence-BERT model (all-MiniLM-L6-v2). Vectors are L2-normalized and stored as float16.
3. Weekly Topic Tracking:
   • Events are bucketed by ISO week.
   • Clustering: K-means clustering is applied within each week to identify dominant themes. The number of clusters ($K$) is determined automatically via the elbow method.
   • Matching: Clusters from adjacent weeks are linked using a greedy one-to-one assignment based on centroid cosine similarity.
   • Labeling: Based on size changes and centroid movement, topics are assigned labels: Emergence, Growth, Decay, Drift, or Stable.
4. Time-Aware Retrieval:
   • As-of Queries: The system filters the knowledge base to include only documents up to a specified cutoff date.
   • Recency Queries: Candidates are re-ranked using a fused score that combines semantic similarity with a temporal decay prior:
     $$\text{score}(q, d, t) = \alpha \cos(q, d) + (1 - \alpha) \cdot 0.5^{\text{age\_days}(t)/h}$$
     Where $\alpha$ balances semantics vs. recency (default 0.7) and $h$ is the half-life in days (default 14). Crucially, the authors note that for freshness to work, the candidate pool must be broad (or the full corpus); a narrow top-$K$ cosine cutoff often discards the newest relevant item before the recency prior can act.

Key Contributions

The paper makes two primary contributions, empirically decoupling the problem of temporal RAG into two distinct challenges:

1. A Tractable Freshness Solution: The authors demonstrate that a simple recency prior (half-life decay) effectively surfaces the most recent relevant evidence. This is achieved by fusing semantic scores with a temporal decay function, allowing newer items to outrank older ones when appropriate.
2. Localization of Heuristic Failure in Topic Evolution: The paper investigates the performance of simple heuristic trackers for topic evolution. It finds that the poor performance of such trackers (low macro-F1) is not driven by the clustering algorithm (e.g., K-means vs. HDBSCAN) but rather by the labeling rule. The rule comparing consecutive weeks (per-week-delta) fails to capture multi-week trends, whereas a baseline-relative rule significantly improves performance.

Experimental Results

The system was evaluated on three corpora of increasing realism:

• Synthetic Log Stream:
  • Freshness: The fused score achieved 1.00 Latest@10 (newest relevant item in top 10), while the cosine-only baseline scored 0.00.
  • Trend Detection: The baseline labeling rule achieved a macro-F1 of 0.08. Replacing the rule with a baseline-relative approach raised this to 0.49, and using ground-truth volumes (removing clustering noise) reached 0.96. Swapping K-means for HDBSCAN had negligible impact (0.10), confirming the rule is the bottleneck.
• CERT Logon Corpus (849k events):
  • This dataset served as a stress test for freshness but was "degenerate" for trend detection due to near-constant text representations.
  • Freshness: With corpus-tuned parameters ($\alpha=0.3, h=3$), the system achieved 1.00 Latest-Set@10. The default synthetic parameters failed (0.00), highlighting parameter sensitivity.
• NVD CVE Descriptions (Diverse Real Data):
  • This corpus tested the "freshest $\neq$ most similar" scenario. The newest relevant CVE was often ranked low by cosine similarity (e.g., 87th percentile for SQL injection).
  • Freshness: The fused score lifted Latest@10 from 0.00 (cosine) and 0.20 (semantic-then-newest) to 0.60 (with tuned $\alpha=0.3$).
  • Relevance: Top-10 relevance (nDCG@10) remained high ($\ge 0.90$), though Precision@10 saw a modest trade-off as recency weight increased.

Significance and Claims

The paper claims to provide a reproducible, decoupled framework for temporal RAG that offers "honest, real-data scope" for each component:

• Freshness is Real but Partial: A simple recency prior is effective for surfacing recent evidence but is parameter-sensitive and dependent on corpus density. It is not a "solved" problem; closing the gap on diverse data requires learned-to-rank models.
• Topic Evolution is a Labeling Problem: The failure of simple heuristic trackers is primarily due to the labeling rule, not the clustering algorithm. This reframes the research direction toward change-aware labeling rules and density-based clustering (to avoid fixed $K$) rather than simply seeking more powerful clusterers.
• Practical Engineering: The approach offers a pragmatic solution for practitioners who cannot retrain base models. It occupies a distinct position in the complexity spectrum: a post-processing step that is more integrated than simple filtering but less complex than full pipeline redesigns (e.g., Temporal Knowledge Graphs or specialized retrievers).

The authors conclude that while basic freshness can be partially addressed with a lightweight temporal prior, achieving deep temporal understanding for AI remains a significant, open research challenge.