FinAbstain: Uncertainty-Calibrated Multimodal RAG for Selective Financial Forecasting

Technical Summary: FinAbstain

Problem Statement
Financial forecasting is inherently difficult due to the asynchronous arrival of evidence and the frequent divergence of signals (e.g., earnings filings vs. news sentiment vs. technical indicators). While Large Language Models (LLMs) can synthesize financial narratives, they often exhibit high confidence even when evidence is sparse, stale, or contradictory. In a decision-making context, this overconfidence converts epistemic limitations into avoidable trading risks. Existing Retrieval-Augmented Generation (RAG) and financial language models suffer from three critical gaps:

1. Temporal Leakage: Standard RAG does not guarantee that retrieved documents were public at the specific forecast timestamp, leading to backtest contamination.
2. Hidden Disagreement: Single synthesis narratives often collapse conflicting signals from fundamental, news, and market sources into a single output.
3. Lack of Selective Refusal: Confidence scores are rarely calibrated to support the operational option of refusing a forecast (abstention) when uncertainty is high.

Methodology: The FinAbstain Framework
FinAbstain is a research framework for uncertainty-calibrated, multimodal RAG with selective prediction. Its architecture consists of four core components:

1. Point-in-Time Multimodal Retriever:
   The system ingests filings (10-K, 10-Q, 8-K), earnings calls, news, prices, volume, and technical indicators. Crucially, it enforces strict temporal constraints: only documents with publication ($\tau_{pub}$) and ingestion ($\tau_{ing}$) timestamps prior to the forecast time $t$ are admitted. This prevents future information leakage. The retriever uses a hybrid dense/BM25 score for text and regime/recency matching for time series, allocating a top-$k$ budget across modalities to prevent news volume from crowding out filings.

2. Role-Separated Agent Protocol:
   Evidence is processed by independent agents before aggregation:

   • Fundamental Agent: Analyzes revenue, margins, cash flow, and guidance.
   • News Agent: Estimates event polarity, novelty, and source diversity.
   • Technical Agent: Analyzes trends, momentum, and volatility using only numerical features.
   • Risk Agent: Constructs counterarguments and identifies missing evidence.
   • Verifier: Checks claims against retrieved spans and detects cross-source contradictions.
     Agents operate independently to reduce anchoring, and a second aggregation pass uses reliability weights without introducing uncited facts.

3. Hybrid Uncertainty Formulation:
   The system aggregates probabilistic assessments from agents and calculates a composite uncertainty score ($U_{i,t}$). This score integrates:

   • Disagreement: Mean Jensen–Shannon divergence between agents.
   • Contradiction: Relevance-weighted probability of contradictory claims.
   • Consistency: Variance across repeated inference samples.
   • Retrieval Deficiency: Lack of supporting evidence.
   • Entropy and Calibration Gap: Predictive entropy and historical calibration errors.
     These factors are combined via a logistic function with validation-fitted weights.

4. Selective Controller:
   The controller predicts a direction (bullish, bearish, neutral) only if the uncertainty score $U_{i,t}$ is below a validated threshold $\theta$. Otherwise, the system abstains. Abstention is treated as a decision, triggering actions such as requesting more evidence, reducing position size, or routing to human review. The threshold is optimized on validation data to minimize selective risk subject to a minimum coverage constraint.

Key Contributions
The paper outlines four specific contributions:

• A temporal RAG specification that enforces document availability at the forecast timestamp for filings, news, and market data.
• A multi-agent evidence protocol that explicitly exposes disagreement, contradiction, and citation support rather than collapsing them into a single narrative.
• A hybrid uncertainty score and formal controller that link calibration, coverage, selective risk, and position sizing.
• A reproducible chronological evaluation blueprint, explicitly distinguishing between simulated planning values and empirical findings.

Results and Evaluation
The paper presents simulated illustrative results rather than empirical findings from a completed backtest. These simulations are designed to test the reporting pipeline and consistency of the proposed metrics.

• Performance: In the simulation, FinAbstain achieves the best calibration (ECE: 0.026) and selective accuracy (68.8%) while covering 72% of cases.
• Risk-Coverage Trade-off: The simulated risk-coverage curves show that FinAbstain maintains lower selective error than baselines (including standard RAG and uncalibrated multi-agent models) at equal coverage levels.
• Ablation: Removing temporal filtering results in deceptively high accuracy (70.4%) in the simulation, which the authors explicitly label as invalid due to future contamination. Removing calibration or abstention capabilities degrades performance, highlighting the necessity of the selective mechanism.
• Metrics: The evaluation framework includes accuracy, F1, Brier score, Expected Calibration Error (ECE), Sharpe ratio, Maximum Drawdown (MDD), and transaction-cost-adjusted returns.

Significance and Claims
The paper positions FinAbstain not as a proven trading strategy, but as a time-safe architecture and a reproducible evaluation blueprint for evidence-grounded selective financial forecasting.

• Core Hypothesis: The authors propose that calibrated abstention may trade coverage for lower selective error and reduced drawdown.
• Modesty of Claims: The authors explicitly state that the reported numbers are "simulated artifacts" for testing methodology and layout. They do not claim economic benefit or empirical validation.
• Future Work: The paper calls for a preregistered chronological study with a frozen data manifest, executable feature pipelines, and independent audits to replace simulated values with empirical results. It acknowledges that the framework cannot eliminate irreducible market uncertainty or strategic misinformation.

In summary, FinAbstain offers a structural approach to financial forecasting that prioritizes epistemic humility (knowing when not to predict) and temporal integrity (ensuring no future leakage), providing a rigorous framework for evaluating when LLMs should abstain from making financial decisions.