Streaming Knowledge Compilation: Proactive Materiality-Scored Pinning for Time-Evolving LLM Wikis

Technical Summary: Streaming Knowledge Compilation for Time-Evolving LLM Wikis

1. Problem Formulation

The paper addresses the limitation of existing LLM wiki systems, which compile knowledge into pre-filled KV caches for efficient inference but assume a static corpus. In high-value domains like finance and real-time encyclopedias, information landscapes evolve continuously. The fundamental challenge is Streaming Knowledge Compilation: given a continuous stream of documents, a fixed token budget, and future queries that are unknown at ingestion time, how does one maintain a compiled wiki that minimizes cumulative regret against an offline oracle with perfect foresight?

The core difficulty lies in deciding which of the hundreds of daily documents warrant incorporation within a fixed budget without waiting for a query to expose a knowledge gap. The paper formalizes this as a budget-constrained online optimization problem where the goal is to minimize regret defined as the difference in answer quality between the online wiki and an oracle wiki compiled with perfect knowledge of future high-materiality events.

2. Methodology: Online WiCER

The authors introduce Online WiCER, an algorithm that transforms the reactive "Compile–Evaluate–Refine" loop of the original WiCER (inspired by Counterexample-Guided Abstraction Refinement, CEGAR) into a proactive, prediction-driven process.

Key Components:

• Materiality Signal ($\phi_t$): The framework relies on a domain-specific signal $\phi_t(k, n) \in [0, 1]$ that scores the informational importance of a document $n$ for an entity $k$ at time $t$. This signal acts as a surrogate for query relevance, enabling proactive pinning before queries arrive.
  • Finance Instantiation: Abnormal stock volatility (Abnormal Volatility Ratio, AVR), defined as the ratio of a stock's 5-day forward realized volatility to the cross-sectional market average.
  • Wikipedia Instantiation: Abnormal Edit Ratio (AER), defined as the ratio of an entity's edit velocity to the cross-sectional average.
• State-Aware Marginal Regret Scorer: Instead of scoring facts in isolation, the system uses a regression head on a frozen LLM backbone to estimate the marginal expected regret reduction of a candidate fact given the current set of pinned facts. This accounts for redundancy; if a fact is already covered by existing pins, its marginal value drops.
• Pin Selection and Eviction:
  • Selection: A greedy algorithm selects facts that maximize marginal regret reduction under the token budget. Theoretically, this achieves a $(1 - 1/e)$ approximation to the optimal set if wiki quality is submodular.
  • Eviction: A decay-weighted priority queue manages the pin budget. Pins decay over time ($\exp(-\lambda \cdot \Delta t)$), ensuring that stale high-scoring facts are evicted to make room for new events.
• Compilation: The system performs incremental updates daily and full recompilation (using the original WiCER algorithm) at periodic intervals ($T_r$) to integrate pinned facts into the narrative structure.
• Predictive CEGAR: The framework formalizes "Predictive CEGAR," where "counterexamples" (knowledge gaps) are predicted from the stream (e.g., high volatility) rather than discovered reactively through failed QA probes.

Theoretical Guarantees:
The paper proves a cumulative regret bound of $O(\sqrt{T} \log K)$. The bound decomposes into an eviction regret term and a prediction error term ($\epsilon = E[|\phi_t - \hat{\phi}_t|]$). Crucially, the regret bound is domain-agnostic; the only domain-specific quantity is the prediction error of the materiality signal.

3. Experimental Results

The framework was instantiated in two domains: Finance (76k articles, 39 S&P 500 companies) and Wikipedia (25 entities, 50k–200k edits).

Finance Domain:

• Materiality Prediction: A lightweight classification head trained on a frozen Llama 3.1 8B backbone achieved an AUROC of 0.728 on a strict temporal split for predicting abnormal volatility. The model successfully identified articles associated with 1.49$\times$ higher realized forward volatility.
• Scorer Architecture: A hybrid approach (zero-shot prompting + linear classification head) showed that the classification head alone captured the discriminative signal, with the zero-shot path serving primarily as a low-latency router for ambiguous cases.
• End-to-End QA: Evaluation revealed a significant confound: when using an LLM-as-judge (the same backbone) to evaluate answers on post-training facts, the "No Wiki" baseline often scored highest due to the judge's preference for fluent, confident parametric responses over factually grounded but specific wiki content.
• Regret Analysis: Despite the QA score confound, regret analysis on matched pairs showed Online WiCER converging to a cumulative regret of -20.0 (mean -0.12/step) against the oracle. The authors interpret the negative sign not as algorithmic superiority, but as evidence of the LLM-judge confound (the oracle's dense, specific facts were penalized by the judge).

Wikipedia Domain:

• Materiality Signal: The Abnormal Edit Ratio (AER) served as the signal.
• Regret Analysis: In this domain, the LLM-judge confound was absent (the backbone lacked parametric knowledge of specific 2024–2025 edits). Here, richer context consistently improved scores. Online WiCER showed a cumulative regret of +16.0 (mean +0.13/step) against the oracle. The positive sign confirms that the algorithm successfully tracked the oracle, with the gap attributed to the noisier AER signal compared to the finance volatility predictor.
• Comparison: The FIFO (recency-only) baseline performed nearly as well as the Oracle in Wikipedia, suggesting that for high-edit-velocity entities, recency is a strong proxy, though the AER signal provided a principled, domain-agnostic selection mechanism.

Efficiency:
The system demonstrated high throughput on commodity hardware (Apple M4), processing articles at rates exceeding 30,000 articles/hour, which is 63$\times$ the peak stream rate of financial news feeds. The state-aware scorer added negligible latency (single dot product) over the frozen backbone forward pass.

4. Key Contributions

1. Formalization of Streaming Knowledge Compilation: Defining the problem of maintaining a budget-bounded wiki against a streaming corpus with unknown future queries.
2. Online WiCER Algorithm: A proactive pinning algorithm using materiality-scored selection, decay-weighted eviction, and periodic recompilation.
3. State-Aware Marginal Scorer: A regression head on a frozen backbone that conditions on the current pin set to select non-redundant facts, with a $(1-1/e)$ submodular approximation guarantee.
4. Regret Decomposition Theorem: Proving an $O(\sqrt{T} \log K)$ regret bound where the only domain-specific term is the prediction error of the materiality signal.
5. Predictive CEGAR: Extending the CEGAR paradigm from reactive counterexample discovery to proactive prediction of knowledge gaps.
6. Methodological Finding: Demonstrating that "LLM-as-judge" evaluation is confounded on post-training facts, establishing regret analysis on matched pairs as the reliable metric for compiled knowledge systems.

5. Significance and Claims

The paper claims that Streaming Knowledge Compilation is a general problem applicable to any domain where knowledge gaps can be predicted from streaming signals (e.g., finance, clinical literature, legal filings). The significance lies in:

• Proactivity: Shifting from reactive updates (waiting for a query failure) to proactive maintenance (predicting high-value events).
• Domain Agnosticism: The algorithm and theoretical guarantees are independent of the specific materiality signal; the domain expert only needs to provide a signal $\phi_t$.
• Theoretical Grounding: Providing formal convergence guarantees and regret bounds for a system operating under query uncertainty.
• Practical Viability: Demonstrating that a single frozen LLM backbone can serve as both the knowledge compiler and the materiality scorer, achieving high throughput and low latency suitable for real-time streams.

The authors remain modest, noting limitations such as the inherent imperfection of materiality prediction (missing "black swan" events), the potential distortion of cross-sectional normalization during correlated crises, and the current reliance on specific domain signals. They do not claim the system is a trading tool but rather a knowledge curation mechanism.