TiMi: Empower Time Series Transformers with Multimodal Mixture of Experts

Time series forecasting is the science of looking at a sequence of numbers recorded over time to guess what will happen next. It is the engine behind predicting everything from the price of gasoline to the spread of a flu outbreak. For decades, the most successful methods have relied almost entirely on the numbers themselves, searching for patterns in the past to project them into the future. However, the real world is rarely just a stream of numbers. It is a complex system influenced by news headlines, government policies, and unexpected events that often appear as text rather than data points. While humans naturally read a news report about a supply chain disruption and adjust their expectations for future prices, traditional computer models have struggled to do the same. They often treat text as a separate, confusing layer that is difficult to align with the smooth flow of numerical data, leading to predictions that miss the mark when the world changes.

A team of researchers at Tsinghua University has proposed a new way to bridge this gap, introducing a system called TiMi. Instead of trying to force text and numbers into the same rigid format, their approach treats the two as distinct but cooperative guides. The core idea is to use a large language model, a type of artificial intelligence trained on vast amounts of human writing, to read the text and reason about what it means for the future. The model does not just memorize the words; it analyzes them to infer causal factors, such as whether a new policy will cause sales to rise or fall. This reasoning is then passed to a specialized forecasting engine that uses the insight to refine its numerical predictions. The researchers found that by letting the language model act as a guide for the numbers, rather than trying to merge them into a single, messy mix, the system could make significantly more accurate forecasts.

The researchers tested this method on sixteen different real-world datasets, covering diverse fields like agriculture, climate, energy, and public health. In these tests, the new system consistently outperformed existing advanced models, including those that had previously tried to combine text and numbers. The results were particularly striking in scenarios where the text provided a clear reason for a change in the numbers, such as a report on a smartphone recall causing a sharp drop in sales. In these cases, the new system successfully used the text to anticipate the drop, whereas other models often failed to capture the sudden shift. The study suggests that the key to better forecasting lies not in forcing different types of data to look the same, but in allowing each to do what it does best: text provides the story and the cause, while numbers provide the record and the trend.

To make this work, the team designed a modular system that acts like a set of specialized advisors. One part of the system focuses entirely on the historical numbers, learning the long-term rhythms and patterns of the data. Another part focuses on the text, using the language model to extract structured insights about future trends, such as whether the outlook is increasing, decreasing, or staying steady. These two streams of information are then brought together in a way that allows the system to choose the most relevant advice for the specific moment. If the text suggests a major event is coming, the system leans heavily on that insight. If the text is vague or irrelevant, the system relies more on the historical patterns. This flexibility allows the model to handle messy, real-world data where text and numbers might not arrive at the same time or in the same format.

The researchers also explored how well this system works when the data is irregular, which is common in the real world. In many situations, news reports might come out at unpredictable times, or data might be missing for certain days. Traditional models often struggle with these gaps, but the new system handled them with ease. Because it processes the text and the numbers separately before combining their insights, it does not require perfect timing between the two. In tests on irregular datasets, the system reduced prediction errors by nearly thirty percent compared to standard methods. This robustness suggests that the approach is not just a theoretical improvement but a practical tool that can work with the imperfect, asynchronous data found in actual applications.

A crucial part of the study involved understanding how the system makes its decisions. The researchers found that the system naturally grouped similar types of data trends together. For instance, when the system was asked to predict the future of a dataset showing a strong upward trend, it consistently relied on one specific part of its internal logic. When the trend was downward, it switched to a different part. This behavior indicates that the system is not just guessing; it is learning to specialize, assigning different "experts" within its brain to handle different kinds of future scenarios. Furthermore, the study showed that the quality of the text matters. When the researchers fed the system noisy or irrelevant text, the system's performance dipped only slightly, proving that it can filter out the noise and focus on the signal.

The findings challenge the prevailing idea that the best way to combine text and numbers is to fuse them into a single, unified representation. Previous attempts often tried to translate text into a numerical code that could be mixed directly with the time series data, a process that often diluted the meaning of the text. The new approach rejects this fusion in favor of a guided interaction. By keeping the text as a source of causal reasoning and the numbers as the subject of prediction, the system preserves the unique strengths of both. The researchers demonstrated that this method works across a wide variety of domains, from tracking influenza cases to predicting traffic volume, suggesting that the ability to read and reason about text is a universal asset for forecasting.

In the end, the work offers a clearer path forward for artificial intelligence in time series analysis. It shows that the future of forecasting may not lie in building larger, more complex models that try to do everything at once, but in designing systems that know how to listen to different kinds of information. By letting a language model read the news and a forecasting model watch the numbers, and then letting them work together, the system achieves a level of accuracy and adaptability that neither could reach alone. The results suggest that for anyone trying to predict the future of complex systems, the story behind the numbers is just as important as the numbers themselves.