🤖 machine learning

ConceptTS: LLM-Guided Concept Bottlenecks for Interpretable Multivariate Time-Series Forecasting

ConceptTS is an interpretable multivariate time-series forecasting framework that leverages large language models to automatically generate and supervise named, human-readable concepts organized into three complementary bottlenecks, achieving competitive accuracy while providing transparent, intervention-ready predictions.

Original authors: Yichen Jiang, Yueqiao Chen, Dongyu Liu

Published 2026-08-24

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Yichen Jiang, Yueqiao Chen, Dongyu Liu

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of data science, machines have become remarkably good at looking at the past to guess the future. When a computer watches a stream of numbers—like the changing levels of air pollution, the fluctuating price of electricity, or the shifting heart rate of a patient—it can often predict what comes next with startling accuracy. These systems are powerful because they can spot complex patterns that humans might miss, weaving together dozens of different variables that change over time. However, there is a significant catch: these powerful machines are often opaque. They produce a prediction, but they do not explain why. It is like asking a weather forecaster for a storm warning, and they simply point to a number without telling you if the danger comes from high winds, heavy rain, or a sudden drop in pressure. In high-stakes situations, such as managing a city's air quality or monitoring a patient's health, knowing the "why" is just as important as knowing the "what." Practitioners need to understand the reasoning behind a forecast to trust it, to spot potential errors, and to know how to intervene if things go wrong.

This is the challenge that a new approach called ConceptTS aims to solve. Researchers have developed a system that forces a machine learning model to think in human-readable ideas rather than just abstract numbers. Instead of letting the computer work in a "black box," this new framework asks the model to identify specific, named concepts that describe what is happening in the data. For example, rather than just crunching numbers, the model might recognize that the wind speed is rising, that the temperature is dropping, or that a specific pollutant is behaving in a way that usually precedes a spike in pollution. To make this possible without requiring humans to manually label every single moment in the data, the researchers used a large language model—an advanced type of artificial intelligence trained on vast amounts of text—to act as a guide. This language model reads the data's history and suggests relevant concepts, then writes a set of rules that the forecasting system can follow to check if those concepts are present.

The system organizes these ideas into three distinct layers to build a complete picture. First, it looks at the recent past to understand the context, asking questions like, "Was the air generally hot or cold in the hours leading up to this moment?" Second, it breaks the future forecast into small chunks, checking for local details like, "Is the wind speed rising in this specific hour?" Finally, it looks at the entire forecast period to spot big-picture trends, such as, "Is the whole upcoming day likely to see extreme temperature swings?" By forcing the model to predict the activation of these specific concepts before it makes its final prediction, the researchers create a transparent path. The model must first decide which concepts are active, and then it uses those decisions to generate the forecast. This means that if the model predicts a spike in pollution, a human can look at the concepts and see that the model was reacting to a combination of low wind speed and high humidity, rather than guessing blindly.

To test this idea, the researchers applied their system to real-world data from Beijing, using air quality measurements from three different monitoring stations over several years. They focused on predicting the concentration of PM2.5, a fine particulate pollutant that poses serious health risks. The results showed that the system could predict the future air quality just as accurately as the most advanced, opaque models currently available. In fact, on one of the test sites, the new system achieved an average error of just 12.71 micrograms per cubic meter, a figure that matched or beat several other top-tier forecasting tools. More importantly, the system provided a clear window into its own thinking. When the researchers examined the concepts the model activated, they found that the machine was correctly identifying real-world conditions, such as cold, dry winter blocks or periods where wind speed and pollution levels moved in opposite directions.

The power of this approach was further demonstrated through a series of experiments where the researchers deliberately interfered with the model's thinking. In one test, they took the model's predictions and forced it to use the exact opposite of the correct concepts. Instead of telling the model that the wind was calm, they told it the wind was blowing hard, even when the data showed otherwise. This simple switch caused the model's predictions to collapse, with the error rate jumping from a manageable 15.37 to a massive 99.66. This dramatic failure proved that the model was not just using the concepts as a side note; it was relying on them as the primary engine for its predictions. The system was genuinely using the human-readable ideas to do its work.

The researchers also explored how much they could trade off between accuracy and transparency. They found that by adjusting a specific setting, they could make the model rely more heavily on the concepts, even if it meant the predictions became slightly less precise. This flexibility is valuable because it allows users to choose how much they want to see inside the machine. In some cases, a slight drop in accuracy might be worth the gain in understanding, especially when the goal is to learn from the data or to explain a decision to a human audience. The system also showed that it could learn complex relationships, such as the fact that when wind speed stays near zero for a long time, pollution levels tend to rise, and as the wind picks up, the pollution clears away. The model learned this pattern and could explain it in plain language, something that traditional black-box models cannot do.

Ultimately, this work represents a shift in how we build intelligent systems for the future. It moves away from the idea that we must sacrifice understanding for accuracy. By using a language model to generate the rules and concepts automatically, the researchers removed the heavy burden of manual labeling that usually makes this kind of transparency impossible. The system does not need a human to sit down and label every hour of data; it learns to recognize the patterns on its own, guided by the language model's suggestions. The result is a forecasting tool that is both powerful and honest, capable of making precise predictions while explaining its reasoning in terms that a human can understand. This opens the door for using advanced artificial intelligence in fields where trust and clarity are essential, ensuring that when a machine predicts the future, we know exactly why it thinks that way.

Technical Summary: ConceptTS

Problem Statement

Multivariate time-series forecasting models, particularly modern architectures like Transformers and CNNs, achieve high accuracy in capturing complex temporal and cross-variable dependencies. However, they suffer from a lack of interpretability; their internal representations are opaque, offering little insight into the specific temporal conditions or variable interactions driving a prediction. This opacity is a critical barrier in high-stakes domains (e.g., healthcare, environmental monitoring) where practitioners must understand failure modes and assess the plausibility of evidence before trusting a forecast. Existing interpretability methods, such as post-hoc saliency maps or prototype-based reasoning, often fail to provide semantically explicit, named evidence that can be directly inspected or intervened upon without incurring prohibitive manual annotation costs.

Methodology: ConceptTS

ConceptTS is an interpretable forecasting framework that organizes predictions around named, human-readable concepts. It bridges the gap between the accuracy of black-box models and the transparency of concept-based models by leveraging Large Language Models (LLMs) for automated concept construction and supervision.

1. LLM-Guided Concept Construction

Instead of relying on manual expert annotation, ConceptTS employs an offline LLM to generate the entire concept pipeline:

Input: The LLM receives dataset statistics, channel descriptions, and cluster profiles derived from training data (e.g., means, slopes, FFT-derived periods, correlations).
Output: The LLM proposes three distinct sets of concepts, each paired with an executable Python predicate:
- Look-back Concepts: Describe semantic patterns in the historical window (e.g., "is history rainy?").
- Forecast Sub-window Concepts: Capture fine-grained local details within specific intervals of the forecast horizon (e.g., "is wind speed rising?").
- Global Forecast Concepts: Describe macro trends over the entire forecast horizon (e.g., "is there an extreme temperature change?").
Labeling: The executable predicates automatically label every training segment, generating binary ground-truth concept activations ( $y_i \in \{0, 1\}$ ) without further LLM calls during training.

2. Architecture

The model follows a structure-preserving encoder-decoder design with parallel Concept Embedding Bottlenecks (CEM):

Encoder: Uses a ModernTCN-style architecture with variable-independent embeddings and large-kernel depthwise convolutions to preserve channel identity and capture broad temporal context. Two parallel encoders process the look-back window and the forecast horizon (in future-aware settings) separately to maintain an information barrier.
Concept Bottlenecks: Three parallel CEM modules process the encoder outputs:
- Look-back CEM: Summarizes historical context.
- Sub-window CEMs: $N_{sw}$ modules capture local patterns for each forecast sub-window.
- Global Forecast CEM: Captures macro trends.
- Mechanism: Each module uses attention-weighted pooling to generate context vectors. It employs a Concept Embedding Model (CEM) structure where concepts have learned static embeddings for active ( $e^+_c$ ) and inactive ( $e^-_c$ ) states. The final concept context is a weighted mixture based on the predicted activation probability $p_c(z)$ .
Decoder: A shared residual sub-decoder constructs the final forecast. It concatenates:
- Concept context vectors from all three bottleneck types.
- Cross-sub-window context (via a BiGRU processing sub-window summaries).
- Positional embeddings.
- A regularized residual channel: A narrow bypass that carries continuous magnitude information not fully captured by discrete concepts. This channel is regularized to control the trade-off between accuracy and interpretability.

3. Training Objective

The model is trained end-to-end with a hybrid loss function:

Forecast Loss: L1 loss on the target variable.
Concept Supervision: Binary Cross-Entropy (BCE) losses forcing predicted activations to match the LLM-derived labels for all three concept sets.
Regularization: Includes boundary smoothness penalties, a variance floor to prevent concept collapse, and an L2 regularizer on the residual channel ( $\alpha_{residual}$ ) to enforce reliance on the concept pathway.

Key Contributions

LLM-Guided Concept Construction: A pipeline that converts domain context and dataset summaries into named temporal concepts and executable labeling rules, eliminating the need for manual per-segment annotation.
Multiscale, Intervention-Ready Architecture: A novel design organizing concepts across historical, local, and global scopes. The architecture preserves temporal/variable structure while limiting information leakage around the bottlenecks, enabling direct concept-level interventions.
Comprehensive Evaluation: Experiments on the Beijing Multi-Site Air Quality dataset demonstrate that ConceptTS achieves accuracy competitive with strong black-box baselines while providing semantically meaningful concept activations.

Experimental Results

The model was evaluated on the Beijing Multi-Site Air Quality dataset (PM2.5 target) across three monitoring sites (Aotizhongxin, Dingling, Tiantan) under two settings:

Future-Aware: Exogenous features (covariates) are known for the forecast horizon.
Look-back-Only: Only historical data is available.

Performance:

In the future-aware setting, ConceptTS achieved an MAE of 12.71 µg/m³ (Aotizhongxin), outperforming Informer (28.62), DeepAR (13.97), and NHiT (16.94), and matching state-of-the-art models like XGBoost (12.34) and TFT (12.44).
In the look-back-only setting, ConceptTS achieved an MAE of 55.59 µg/m³, outperforming TimeXer (57.43) and iTransformer (59.16), and remaining competitive with the best baselines (within ~0.55–0.81 µg/m³ of the top performers).

Interpretability & Ablation:

Concept Reliance: Replacing predicted concept activations with inverted ground-truth labels caused the test MAE to degrade from 15.371 to 99.661, providing direct evidence that the decoder relies heavily on the concept-activation channel.
Accuracy-Interpretability Trade-off: Adjusting the residual channel weight ( $\alpha_{residual}$ ) allowed for a controlled trade-off. Lower weights improved accuracy but reduced reliance on concepts; higher weights forced the model to route predictions through the concept bottlenecks, with concept-only predictions still capturing the correct macro/micro shapes of the ground truth.
Intervention: Counterfactual sweeps (e.g., changing "is calm block" from 0 to 1) showed that increasing the activation of calm-air concepts consistently raised predicted pollution levels, aligning with physical intuition.

Significance

The paper claims that ConceptTS addresses the critical need for transparent multivariate time-series forecasting without sacrificing predictive performance. By automating the discovery of semantically explicit concepts via LLMs, it overcomes the annotation bottleneck of traditional Concept Bottleneck Models (CBMs). The framework provides a stable semantic vocabulary for inspecting model reasoning and allows for direct interventions, making it suitable for high-stakes applications where understanding the "why" behind a forecast is as important as the forecast itself. The authors note that while the model handles typical timesteps well, it is slightly less sensitive to extreme pollution spikes compared to some baselines, a limitation identified for future work.