Technical Summary: Dual-Stream Multimodal Learning for Long-Horizon Influenza Forecasting

1. Problem Definition

The paper addresses the challenge of forecasting long-range influenza-like illness (ILI) activity, specifically predicting 12 weeks of future ILI trends based on a 36-week historical window. The core difficulty lies in effectively fusing two distinct data modalities available at the same forecast origin:

1. Numerical Signals: Structured, weekly epidemiological surveillance data (e.g., ILI percentages across age groups).
2. Textual Signals: Noisy, loosely structured weekly news headlines and search summaries describing symptoms, circulating viruses, and public health observations.

Existing approaches often treat these modalities as fixed features, concatenating them after independent encoding or using text as a one-way conditioning signal. This fails to explicitly model the dynamic interaction where the state of one modality should influence the interpretation of the other (e.g., using text to contextualize a numerical spike, or using numerical trends to determine the epidemiological relevance of a textual signal). Furthermore, textual signals are often lagged or indirectly related to near-term trends, requiring careful architectural design to avoid noise propagation.

2. Methodology: Dual-Stream Attention (DSA)

The authors propose Dual-Stream Attention (DSA), a multimodal deep learning framework designed to let numerical and textual streams condition each other before temporal aggregation. The architecture consists of five key components:

2.1. Numerical Encoder

The numerical history ($\mathbf{X}_{num} \in \mathbb{R}^{36 \times 14}$) is projected to a hidden size and processed by a 4-layer Transformer encoder. Notably, the authors omit positional encoding at this stage, relying instead on the causal masking applied later in the temporal model to preserve order, as adding positional encoding early degraded performance in their experiments.

2.2. Headline Encoder

Weekly text snippets are tokenized and encoded using BioLinkBERT-base, a language model pretrained on biomedical text.

• Adaptation: Only the last 8 layers are unfrozen during training to balance adaptation with overfitting.
• Pooling: Instead of using a [CLS] token, the model employs masked mean pooling over token representations to generate weekly embeddings ($\mathbf{e}_t$).
• Inter-week Attention: These embeddings pass through a self-attention layer to allow the text representation at week $t$ to incorporate context from other weeks before fusion.

2.3. Cross-Modal Attention (CMA) Module

This is the core innovation of DSA. Rather than simple concatenation, CMA implements a bidirectional attention mechanism:

• Numerical $\to$ Text: The numerical stream queries the text stream to interpret how the text should be read given the current epidemiological state.
• Text $\to$ Numerical: The text stream queries the numerical stream to interpret how the numerical trend should be read given the textual context.
• Fusion: The outputs of both attention directions are concatenated, projected, and added back to the normalized streams via residual connections, followed by Layer Normalization. This allows each modality's representation to be reweighted based on the other.

2.4. Causal Temporal Model

The fused representation is passed through a 2-layer causal Transformer. A strict attention mask ensures that the model can only attend to current and past positions within the lookback window, preventing information leakage from future time steps. The final hidden state of the last time step is pooled for forecasting.

2.5. Regressor

A two-layer Multi-Layer Perceptron (MLP) maps the final temporal representation directly to a 12-week forecast horizon. The model is trained end-to-end using Mean Squared Error (MSE).

3. Key Contributions

1. Architecture: Introduction of DSA, which explicitly models bidirectional cross-modal conditioning between numerical epidemiological history and concurrent weekly text, moving beyond unconditional concatenation.
2. Rigorous Evaluation: A multi-seed evaluation (10 random seeds) against strong unimodal (iTransformer, DLinear, PatchTST), text-augmented (TaTS), and LLM-based (GPT4MTS) baselines. The study includes analyses of tail-risk, horizon-wise performance, phase-stratified robustness, and Model Confidence Sets.
3. Ablation Analysis: Controlled experiments demonstrating that the performance gain stems primarily from the bidirectional fusion mechanism rather than language model fine-tuning or specific encoder choices.
4. Faithfulness Analysis: Perturbation-based masking analysis showing that the learned cross-modal attention is functionally informative, with a stronger effect observed in the Text-to-Numerical direction.

4. Experimental Results

Evaluated on the Time-MMD Health_US dataset (36-week lookback, 12-week forecast):

• Accuracy: DSA achieved a median test MSE of 0.416, significantly outperforming iTransformer (0.668), TaTS (0.607), and GPT4MTS (0.851). This represents mean-error reductions of 54.95%, 37.29%, and 67.23% respectively.
• Robustness & Tail-Risk: DSA demonstrated substantially lower worst-window error (Max MSE) and P90 MSE compared to all baselines. It ranked first in 100% of bootstrap draws, whereas GPT4MTS ranked last in 74.6% of draws despite competitive average errors.
• Phase-Stratified Performance:
  • Peak Phase: DSA achieved the clearest margin over all baselines during the epidemic peak, the most operationally critical regime.
  • Other Phases: DSA remained competitive in off-season, rising, and declining phases, though it was slightly outperformed by TaTS in the rising phase and GPT4MTS in the declining phase.
• Generalization: On an external dataset (Time-MMD Health_Africa), DSA again ranked first among nine evaluated baselines.
• Ablations:
  • Removing the causal mask increased MSE significantly (1.09 $\to$ 1.71).
  • Freezing the text encoder (no fine-tuning) still outperformed the best fusion baseline (TaTS), indicating the fusion mechanism is the primary driver of improvement.
  • Bidirectional attention outperformed unidirectional variants (Text$\to$Num or Num$\to$Text alone).
  • Replacing BioLinkBERT with general-domain BERT or smaller models (MiniLM) caused a slight drop in performance but did not collapse the model's effectiveness, suggesting the advantage relies more on the fusion architecture than specific pretraining.

5. Significance and Claims

The paper claims that DSA offers a robust overall architecture for long-horizon influenza forecasting rather than a model that dominates under every specific epidemiological condition. Its primary significance lies in:

• Explicit Interaction: Successfully modeling how noisy, temporally offset surveillance text should inform numerical trends without being trusted unconditionally.
• Operational Value: By maintaining consistent competitiveness across all epidemic phases (especially the critical peak phase) and minimizing tail-risk (worst-case errors), DSA provides more practical value for public health resource allocation than models that excel only in specific regimes or average cases.
• Methodological Insight: Demonstrating that bidirectional cross-modal attention is superior to unidirectional conditioning or simple concatenation in multimodal time-series forecasting, particularly when modalities have different reliability and temporal scales.

The authors conclude that while DSA is not universally dominant in every single phase (e.g., declining phase), its consistency across random seeds, phases, and geographies makes it a superior choice for deployed forecasting systems where the future epidemiological phase is unknown.