🤖 machine learning

Interpretability in Deep Time Series Models Demands Semantic Alignment

This paper argues that interpretability in deep time series models requires "semantic alignment," where predictions are expressed through user-meaningful variables and mechanisms that respect temporal constraints, and proposes a formal definition and design blueprint to achieve this goal.

Original authors: Giovanni De Felice, Riccardo D'Elia, Alberto Termine, Pietro Barbiero, Giuseppe Marra, Silvia Santini

Published 2026-06-02

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Giovanni De Felice, Riccardo D'Elia, Alberto Termine, Pietro Barbiero, Giuseppe Marra, Silvia Santini

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Core Problem: The "Black Box" vs. The "Human Mind"

Imagine you are an engineer watching a machine. You see a warning light flash. You know why it flashed: the machine got too hot, the metal expanded, and a part started to wear down. You think in terms of heat, stress, and wear.

Now, imagine a super-smart AI computer is also watching that same machine. It predicts the machine will break in 10 minutes. But when you ask the AI why, it says: "Because the value of 'Hidden Unit 47' at 'Time Step 302' crossed a specific threshold."

This is the problem the paper addresses. Current AI models for time series (data that changes over time, like stock prices, heart rates, or machine sensors) are incredibly accurate at predicting the future, but they are opaque. They speak a language of math and internal numbers that humans don't understand.

The authors argue that existing methods to "explain" these models are like translating a book into a different language that is still gibberish to the reader. They show how the computer calculated the answer (the math), but they don't show why the answer makes sense to a human expert.

The Solution: "Semantic Alignment"

The paper proposes a new rule for building these AI models called Semantic Alignment.

The Analogy: The Translator vs. The Co-Pilot

Current Approach (The Translator): The AI does its work in its own secret language. Afterward, a translator tries to explain the result to you. But the translator might say, "The AI thought about 'Vector X'," which doesn't help you make a decision.
The Paper's Approach (The Co-Pilot): The AI is built from the ground up to think in your language. Instead of "Vector X," it thinks in terms of "Overheating" or "Thermal Stress." It doesn't just translate the answer at the end; it uses human concepts as the actual building blocks of its thinking process.

What "Semantic Alignment" Means:

Variables must match: The internal numbers the AI uses must represent things humans care about (e.g., "Temperature" instead of "Activation Layer 5").
Mechanisms must match: The way the AI connects these ideas must follow rules humans understand (e.g., "If heat goes up, stress goes up").
Time matters: This is the paper's unique twist. In static pictures (like identifying a cat in a photo), you just need to match the concepts. But in time series, things change. The paper argues that the AI must not only use human concepts but also ensure those concepts evolve over time in a way that makes sense to humans. If the AI predicts "stress" today, it must predict "stress" tomorrow in a way that is consistent with how stress actually accumulates in the real world.

The Blueprint: How to Build These Models

The authors provide a "blueprint" for building these aligned models. Think of it like building a house with specific, transparent rooms instead of a maze of hidden corridors.

The Encoder (The Translator): The model looks at raw data (like a stream of temperature numbers) and immediately converts it into human concepts (e.g., "Is it overheating?").
The Propagation (The Storyteller): The model takes those concepts and moves them forward in time. It uses specific rules to update them. For example, if "overheating" happens, "wear and tear" should increase. This step ensures the story the AI tells remains logical as time passes.
The Decoder (The Answer): Finally, the model uses these human concepts to make a prediction (e.g., "The machine will break").

How to make sure it works:
The paper suggests training the model with three goals at once:

Task Loss: Make sure the final prediction is accurate.
Concept Loss: Make sure the AI correctly identified the "overheating" or "stress" concepts.
Propagation Loss: Make sure the AI correctly updated those concepts over time (e.g., ensuring stress doesn't magically disappear).

Why This Matters (The Benefits)

If we build models this way, we get several superpowers that current "black box" models don't have:

Actionable: If the AI says "The machine will break because of high stress," an engineer can actually do something about it (cool the machine down). If the AI says "Hidden Unit 47 is high," the engineer is stuck.
Verifiable: We can check the math. We can say, "Does this model follow the law of physics where stress increases with heat?" If the model is aligned, we can prove it. If it's a black box, we can't.
Trustworthy: In high-stakes fields like healthcare or finance, you can't just trust a number. You need to understand the reasoning. If the reasoning uses concepts you understand, you can trust the result.

What the Paper Says About Common Objections

The authors address four common reasons people might not want to do this:

"Just explain it after the fact (Post-hoc)."
- Rebuttal: Explaining a black box after it's done is like trying to guess a chef's recipe by tasting the soup. You might get close, but you can't be sure, and you can't change the recipe if you don't like it. The model needs to be built with the recipe in mind from the start.
"Just show us the math (Mechanistic Transparency)."
- Rebuttal: Showing the math is like showing someone a blueprint of a car engine in a language they don't speak. It's transparent, but not understandable. You need the concepts (the engine, the fuel) to be named correctly.
"It will make the AI less accurate."
- Rebuttal: The authors argue this is a myth. You can build models that are both human-readable and super accurate. You can even add "secret backdoors" (residual paths) for the AI to use if it needs to be extra precise, while keeping the main path human-readable.
"It's too hard to label the data."
- Rebuttal: Yes, asking humans to label "overheating" is hard. But the paper suggests we can use other AI tools (like Large Language Models) to help label the data, or use rules from physics to guide the model so we don't need as many labels.

Summary

The paper argues that for AI to be truly useful in the real world, especially for data that changes over time, it must stop speaking in "computer code" and start speaking in "human concepts." It must be Semantically Aligned. This means the AI's internal thoughts should be about "heat," "stress," and "time," just like a human expert's thoughts are. This doesn't just make the AI easier to understand; it makes it safer, more trustworthy, and actually useful for making decisions.

Technical Summary: Interpretability in Deep Time Series Models Demands Semantic Alignment

Problem Statement

Deep learning (DL) models have achieved state-of-the-art performance in time series analysis but remain limited in high-stakes domains (e.g., healthcare, finance) due to their "black-box" nature. Existing interpretability approaches primarily focus on structural opacity—explaining internal computations like attention weights or recurrent states. However, the authors argue these methods fail to address semantic opacity: the disconnect between the model's internal variables and the meaningful concepts used by domain experts to reason about the phenomenon.

Current methods often explain how a model computes a result (e.g., "timestep 47 was important") without explaining what that result means in domain terms (e.g., "onset of tachycardia" or "thermal stress accumulation"). Furthermore, existing frameworks lack a formal definition of interpretability that accounts for the temporal evolution of concepts, a critical requirement absent in static settings.

Methodology and Formal Framework

1. Defining Semantic Alignment (SA)

The paper proposes Semantic Alignment (SA) as the necessary condition for interpretability in deep time series models. SA requires that a model's internal variables and inference mechanisms correspond structurally to the concepts and reasoning processes of domain experts.

The authors formalize this using a probabilistic framework:

Concepts ( $C$ ): Human-understandable random variables (e.g., binary labels, real-valued quantities) that can be instantaneous (snapshots) or dynamic (evolving over time).
Mechanisms: Conditional probability distributions (CPDs) describing how concepts depend on one another and evolve.
Alignment Conditions:
1. Variable Alignment: The model's encoder must map raw observations to representations ( $U_t$ ) that align with instantaneous concepts ( $C^U_t$ ), and the propagation module must produce representations ( $Z_{t+1}$ ) that align with dynamic concepts ( $C^Z_{t+1}$ ). Formally, $P(U_t = C^U_t | x_{\le t}) = 1$ and $P(Z_{t+1} = C^Z_{t+1} | x_{\le t}) = 1$ .
2. Mechanism Alignment: The relationships between concepts must satisfy constraints defined by human knowledge (e.g., monotonicity, physical laws). This is framed as a constraint satisfaction problem where the model's mechanism $P(V_{out} | V_{in})$ must belong to a set of admissible distributions $\mathcal{M}^{(h)}$ defined by the user.

2. Blueprint for Semantically Aligned Models

The authors outline a blueprint extending Concept-Based Models (CBMs) to the temporal domain. The architecture consists of three stages:

Concept Encoding: A module maps raw time series windows to source concepts (instantaneous or derived directly from input).
Concept Propagation: A structured mechanism evolves these concepts over time. This includes:
- Temporal mechanisms: How a single concept evolves ( $P(c_{t+1} | c_{\le t})$ ).
- Spatio-temporal mechanisms: How different concepts influence each other over time.
Task Decoding: A decoder maps the stack of propagated concepts to the final task output (optional if the output is itself a concept).

3. Training Objectives

To enforce SA, the paper proposes a composite loss function:
$\mathcal{L} = \alpha \mathcal{L}_{task} + \beta \mathcal{L}_{concept} + \gamma \mathcal{L}_{prop}$

$\mathcal{L}_{task}$ : Standard task supervision.
$\mathcal{L}_{concept}$ : Supervises the encoder to match predicted source concepts with ground-truth labels.
$\mathcal{L}_{prop}$ : Supervises the propagation layers to ensure temporal consistency and alignment of dynamic concepts with future ground-truth labels. This term is crucial for preventing concept drift over time.

Mechanism alignment can be enforced via architectural constraints (e.g., using monotonic layers) or soft constraints (regularization) on the hypothesis space.

Key Contributions

Formal Definition of Semantic Alignment: The paper provides the first mathematical formulation of SA for time series, distinguishing between instantaneous and dynamic concepts and explicitly requiring alignment to be preserved under temporal evolution.
Identification of Semantic Opacity: It distinguishes semantic opacity from structural opacity, arguing that explaining internal computations is insufficient if those computations do not map to human-understandable domain concepts.
A Design Blueprint: It proposes a concrete architectural template (Encoding-Propagation-Decoding) that extends concept-based models to handle temporal dependencies, ensuring that predictions are expressed in terms of meaningful variables.
Analysis of Existing Paradigms: The paper critically evaluates current interpretability methods (post-hoc explanations, attention mechanisms, mechanistic interpretability, physics-informed models) and demonstrates via a taxonomy (Table 1) that most fail to satisfy both dynamic concept alignment and mechanism alignment simultaneously.

Results and Claims

The paper is a position paper and does not present new empirical benchmarks or experimental results comparing specific model performances. Instead, its "results" are theoretical and analytical:

Theoretical Guarantee: The authors argue that their blueprint provides necessary conditions for interpretability that existing methods lack, specifically the preservation of alignment over time.
Feasibility of Accuracy: The paper challenges the assumption that interpretability necessitates a trade-off in accuracy. It cites recent static CBM literature showing that concept-based models can match black-box performance, and suggests that residual pathways and concept embeddings can preserve expressivity in temporal settings.
Scalability of Annotations: It counters the claim that concept annotation is prohibitively expensive by pointing to emerging automated methods (LLM-based annotation, concept discovery tools) and the potential to reduce labeling granularity in time series (e.g., labeling every $n$ -th timestep).

Significance and Impact

The authors claim this work is significant for several reasons:

Trust and Actionability: Semantically aligned models enable verifiability (via formal verification against domain properties) and actionability (allowing humans to intervene directly on concept values or mechanisms, e.g., correcting a clinical state to update future predictions).
Robustness: By constraining behavior at the level of stable, domain-relevant concepts rather than raw observations, these models may be more robust to distribution shifts.
Bridging Domains: The framework acts as a bridge between interpretable models and hybrid neurosymbolic architectures, allowing for the integration of formal constraints and symbolic reasoning into deep time series models.
Paradigm Shift: The paper calls for a shift in the research community from "post-hoc explanation" to "interpretability-by-design," urging researchers to explicitly state which types of alignments (variables and mechanisms) their models guarantee.

In summary, the paper argues that for deep time series models to be safely deployed in critical applications, they must be designed to align structurally with human reasoning, preserving the meaning of concepts as they evolve over time.