🔢 mathematics

When Does Text Inform? Benchmarking Information-Theoretic Metrics for Multimodal Time-Series Forecasting

This paper introduces a synthetic benchmark with known ground-truth information content to evaluate six mutual information estimators for auditing text annotations in multimodal time-series forecasting, demonstrating their ability to identify informative text and select optimal annotations for downstream tasks without model training.

Original authors: Emma Andrews, Gianmarco Mengaldo

Published 2026-09-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Emma Andrews, Gianmarco Mengaldo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of predictive modeling, computers have long been experts at reading numbers. They can track the rise and fall of stock prices, the fluctuation of energy demand, or the spread of a disease by analyzing patterns in historical data. But the real world is rarely just numbers; it is also filled with words. A sudden news report about a supply chain disruption, a doctor's note describing a patient's unusual symptoms, or a meteorologist's comment on an approaching storm can hold clues that raw data alone misses. The promise of modern artificial intelligence is to combine these two streams—numerical signals and natural language—into a single, smarter forecast. The hope is that by feeding a computer both the numbers and the story behind them, the machine can predict the future with greater accuracy.

However, a significant problem has emerged in this field. Just because a text annotation exists alongside a time series does not mean it is helpful. Sometimes, the words describe a future event that the numbers cannot yet see. Other times, the text is simply wrong, or it describes something entirely irrelevant to the specific moment being predicted. In the rush to build more complex systems, researchers often fuse text and data without knowing if the words are actually improving the prediction or just adding noise. Without a way to measure the true value of the text before training a model, practitioners risk wasting resources on misleading information or, worse, building systems that learn to ignore the very clues they were designed to use.

A team of researchers at the National University of Singapore has tackled this uncertainty by creating a controlled environment to test how well we can measure the value of words. They built a synthetic benchmark, a made-up dataset where the truth is known by design. Imagine a simple, repeating wave pattern, like the steady rise and fall of a tide. The researchers then inserted specific moments where the wave suddenly stopped and flattened out, a change the pattern itself could not predict. At these exact moments, they attached three types of text notes. One type correctly described the upcoming flat line. A second type described the exact opposite, claiming the wave would continue to rise or fall. The third type offered generic, vague observations about the wave that provided no clue about what would happen next. Because the researchers created the data, they knew with absolute certainty which notes were helpful, which were harmful, and which were useless.

Using this perfectly labeled ground truth, the team tested six different mathematical tools designed to measure how much information a piece of text provides about a future event. These tools, known as mutual information estimators, are meant to act as a diagnostic check, telling a researcher whether a text annotation is worth the effort of including in a model. The researchers fed the correct, incorrect, and irrelevant notes into these tools to see if the tools could correctly rank them. They found that the tools generally worked well in identifying the helpful notes as the most informative. However, the study revealed that not all tools are created equal. Some of the more complex, neural-network-based tools struggled when the signal from the text was weak, often producing unreliable or negative results. In contrast, simpler, deterministic tools proved more robust, consistently ranking the correct notes highest and the irrelevant notes lowest, even when the text was mixed with noise.

The researchers then took their findings to the real world, testing these same tools on seven different datasets covering agriculture, public health, energy, and finance. Here, the situation was messier. Unlike their synthetic wave, real-world data is noisy and complex, and the text annotations are not always perfectly timed to the events they describe. The study showed that in these real-world scenarios, the text often carried general information about a topic but was not tightly linked to the specific timestamp it was attached to. For instance, a news article about a drought might be relevant to crop yields for months, but a tool might struggle to pinpoint exactly which day the drought would impact the numbers. The researchers discovered that while the text did contain useful information, simply fusing it with the data often made predictions worse rather than better. The text was not always wrong, but it was often too diffuse to help a model predict a specific future value.

Based on these experiments, the team established a set of practical rules for anyone trying to combine text and time series data. They recommend using specific, stable tools to audit text before training a model, rather than relying on complex neural estimators that can be unstable with small datasets. They also suggest that instead of just asking if text is generally useful, researchers should check if the text is actually paired correctly with the specific moment it describes. If the pairing is weak, the text might be better left out entirely. The study concludes that while the idea of combining words and numbers is powerful, it requires a careful, principled approach to ensure that the words are actually informing the forecast and not just confusing the machine. By providing a way to measure the true value of text before a model is even built, this work offers a path toward more reliable and transparent forecasting systems in fields where getting the prediction right matters most.

Technical Summary: When Does Text Inform? Benchmarking Information-Theoretic Metrics for Multi-Modal Time-Series Forecasting

Problem Statement

Multimodal time-series forecasting, which integrates numerical signals with natural language annotations, promises richer predictions by leveraging textual context. However, a critical gap exists in determining whether a specific text annotation meaningfully contributes to forecast accuracy. In practice, text may be correct, incorrect, or irrelevant, yet practitioners lack a principled method to audit annotation quality before committing to expensive model training.

Current approaches rely on evaluating mutual information (MI) estimators against human judgment on real-world datasets. This creates a circular dependency: the "ground truth" of information content is unknown, making it impossible to rigorously validate whether an MI estimator correctly identifies informative text. Furthermore, existing synthetic benchmarks for MI estimation (e.g., Czy˙z et al., 2023) operate on continuous numerical variables and do not account for the semantic structure of natural language or the specific categories of annotation quality (correct, incorrect, irrelevant).

Methodology

1. MMTT-Bench: A Synthetic Oracle Benchmark

To resolve the lack of ground truth, the authors introduce MMTT-Bench (Multimodal Text-Time Series Bench). This is a synthetic dataset constructed with an "oracle" ground truth regarding information content.

Signal Generation: The core signal is a sine wave with randomly inserted constant-value segments (flat lines at $y=-0.5$ ), covering ~30% of the time axis. These flat segments create "transition states" where the time series alone provides no deterministic information about the future, making text the dominant source of predictive value.
Annotation Categories: At each annotation point, three text variants are generated based on the known future state of the signal:
- Correct: Accurately describes the upcoming transition (e.g., "The signal will drop to a constant").
- Incorrect: Describes the opposite of the true future state (e.g., "The signal will rise"). Crucially, 30% of incorrect annotations use the same transition templates as correct ones but are distributed randomly, ensuring they carry statistical dependence (albeit inverted) with the future signal.
- Irrelevant: Describes generic properties of the signal (e.g., periodicity) with no directional or predictive information.
Data Split: The dataset is split chronologically (62.5% train, 18.75% validation, 18.75% test) to prevent leakage.

2. Estimation Framework

The study evaluates six complementary Mutual Information (MI) estimators to measure the unique contribution of text ( $X_{text}$ ) to the future signal ( $Y$ ) given the time series ( $X_{ts}$ ). The primary metric of interest is the conditional MI: $I(X_{text}; Y | X_{ts})$ .

Estimators Evaluated:
1. KSG (K-Nearest-Neighbor): Non-parametric, density-based.
2. MINE (Mutual Information Neural Estimator): Neural critic-based lower bound.
3. InfoNCE: Contrastive learning-based lower bound.
4. CCA (Canonical Correlation Analysis): Gaussian assumption-based, exact under linearity.
5. PID (Partial Information Decomposition): Decomposes joint information into unique, redundant, and synergistic atoms.
6. V-Information: Measures information exploitable by a specific predictive family (Ridge regression), reported as $\Delta R^2$ .
Representation Pipeline: Text is embedded using various models (e.g., DistilRoBERTa) and reduced via PCA. Time series inputs are tokenized. The pipeline is tuned to ensure robust ordering of MI estimates across categories.

3. Validation Strategy

Synthetic Validation: The estimators are tested on MMTT-Bench to see if they correctly rank the categories: Correct > Incorrect > Irrelevant.
Downstream Correlation: The MI estimates are correlated with the Mean Squared Error (MSE) of 14 different model architectures (including Transformers and simple regressors) trained on the respective corpora.
Real-World Validation: The findings are extended to seven domains from Time-MMD and ten stocks from FinTexTS. Since these lack oracle ground truth, the authors use "alignment contrasts" (comparing correctly paired text vs. misaligned text) to assess whether text is tied to specific timestamps.

Key Results

1. Estimator Performance on Synthetic Data

Ranking Accuracy: All six estimators successfully identified Correct text as the most informative.
Differentiation:
- CCA, V-Information, and PID consistently ranked Incorrect > Irrelevant, correctly identifying that incorrect text (which often describes an inverted signal) still carries statistical dependence.
- KSG and PID struggled to distinguish Incorrect from Irrelevant in some configurations, often assigning near-identical values, likely due to the averaging nature of non-neural estimators failing to capture sign-inverted dependencies.
- Neural Estimators (MINE, InfoNCE) were highly sensitive to the inverted signal, detecting the dependence in Incorrect text more strongly than non-neural methods. However, they exhibited high variance and instability on weak signals.
Conditional MI Issues: KSG frequently produced negative conditional MI values when the marginal MI ( $I(X_{ts}; Y)$ ) was large, due to the subtraction of two biased estimates. CCA and V-Information did not exhibit this pathology.

2. Correlation with Downstream Performance

There is a strong positive correlation between the conditional MI estimates and downstream forecasting performance (MSE).
CCA emerged as the most reliable auditor, showing the highest Spearman correlation ( $|\rho| \approx 0.90\text{--}1.00$ ) between MI scores and model performance across various architectures.
The metrics successfully identified that a corpus with 100% correct text yields the best performance, and that even a small contamination (e.g., 25% irrelevant text) degrades performance in a predictable manner.

3. Real-World Findings

Weak Signals: On real-world datasets (Time-MMD, FinTexTS), where the time series itself is often highly predictable (high $R^2$ ), the conditional MI of text is small.
Estimator Failure Modes:
- MINE collapsed to near-zero or negative estimates.
- InfoNCE showed very high variance.
- KSG again produced negative conditional values.
- CCA, V-Information, and PID remained stable and correctly ordered.
Alignment Contrasts: In real-world datasets, text was found to carry measurable information in aggregate but was often not tied to the specific timestamp it was attached to. The "alignment contrast" (difference between correct pairing and misaligned pairing) was statistically indistinguishable from zero for most pairs, suggesting that while the text is topically relevant, it does not provide precise temporal predictive power.

Significance and Recommendations

The paper establishes that information-theoretic metrics can serve as a principled, pre-training diagnostic tool for multimodal time-series forecasting, provided the right estimator is chosen. The authors release MMTT-Bench and all code to enable reproducible evaluation.

Practical Recommendations:

Default Audit Pair: Use CCA and V-Information. They are deterministic, require no training, satisfy the chain rule (avoiding negative conditional MI), and show the highest correlation with downstream performance.
Neural Estimators: Use MINE or InfoNCE only when sample sizes ( $N$ ) are large and expected conditional MI is substantial. They are unreliable on weak signals common in real-world data.
KSG Limitation: Do not use KSG for conditional MI when the marginal information of the time series is large, as it leads to negative estimates.
PID Usage: Use PID only when the target variable has interpretable discrete structure; otherwise, binning choices introduce arbitrary sensitivity.
Paired Contrasts: Prefer comparing conditional MI against a shuffled/misaligned version of the same corpus (alignment contrast) rather than comparing against zero. This cancels estimator bias and helps determine if text is truly timestamp-specific.

The work concludes that while text fusion is promising, its value is highly variable. Reliable pre-training auditing using metrics like CCA can prevent the integration of misleading or irrelevant text, transforming how multimodal systems are built in high-stakes domains like healthcare and finance.