🤖 machine learning

XCTFormer: Leveraging Cross-Channel and Cross-Time Dependencies for Enhanced Time-Series Analysis

XCTFormer is a novel transformer-based model that explicitly captures cross-temporal and cross-channel dependencies through a token-to-token Cross-Relational Attention mechanism, achieving state-of-the-art performance in multivariate time-series tasks, particularly in imputation.

Original authors: Israel Zexer, Omri Azencot

Published 2026-05-19

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Israel Zexer, Omri Azencot

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to predict the weather, but instead of looking at just one thermometer, you have a room full of them, plus barometers, wind sensors, and humidity gauges. In the world of data science, this is called multivariate time-series analysis. The goal is to look at all these different "channels" of information together to understand what's happening and predict the future.

For a long time, experts believed that to make the best predictions, you needed a model that understood how all these sensors talked to each other. If the wind picks up, the temperature might drop; if the humidity rises, rain might follow. These are dependencies.

However, a recent surprise in the field showed that models which ignore these conversations (treating each sensor as if it were alone in a room) often performed better than the ones trying to listen to the whole group. It was like finding that a solo singer often hits the right notes better than a choir that's trying to harmonize.

The authors of this paper, XCTFormer, ask: "Why is the choir failing? Is it because they are singing the wrong notes, or because they are trying to listen to each other in a messy, confusing way?"

They argue that previous "choir" models were trying to understand the relationships indirectly, like trying to hear a conversation in a noisy room by only listening to the volume. They missed the subtle, direct connections.

The Solution: The "Token-to-Token" Super-Listener

The authors introduce XCTFormer, a new model designed to listen to every single piece of data, at every single moment in time, and understand exactly how it relates to every other piece.

Here is how they built it, using some everyday analogies:

1. The Data Processing: Breaking the Story into Scenes

Instead of looking at the whole history of a sensor at once, the model chops the data into small chunks called patches. Think of this like taking a long movie and cutting it into short, manageable scenes. This helps the model focus on local patterns (like a sudden spike in temperature) without getting overwhelmed by the whole movie.

2. The Core Engine: CRAB (The Cross-Relational Attention Block)

This is the brain of the operation. In standard AI models, the "attention" mechanism is like a spotlight that shines on the most important parts of the data. But usually, this spotlight can only shine brighter (positive weights).

XCTFormer's CRAB is special because:

It learns who to ignore: It uses a "learnable mask." Imagine a conductor who doesn't just tell the orchestra to play louder, but also tells specific instruments to play softer or even stop entirely if they are distracting. This helps the model focus on the most crucial relationships.
It understands "negative" relationships: Standard models can only say, "This is important!" (Positive). XCTFormer can also say, "When this goes up, that must go down!" (Negative). It uses a new math trick (called AbsAct) that allows the spotlight to shine in reverse, capturing complex "see-saw" relationships that other models miss.

3. The Scalability Plugin: DeCoP (The Compression Assistant)

There is a catch: If you have 1,000 sensors and 1,000 time steps, checking every single connection between them creates a massive web of connections (1 million connections!). This is too heavy for computers to handle.

To fix this, they added DeCoP. Think of this as a summarizer. Instead of reading every single page of a 1,000-page book to find the connection, DeCoP reads the book and writes a 50-page summary that keeps all the important plot points but throws away the fluff. This allows the model to handle huge datasets without crashing, while still keeping the most important information.

What Did They Find?

The authors tested their new "Super-Listener" on three main tasks:

Forecasting (Predicting the Future): They tried to predict things like electricity usage and traffic. XCTFormer performed very well, often beating the best existing models, especially on a tricky synthetic test designed to see if models could ignore "fake" signals (distractors).
Imputation (Filling in the Blanks): Imagine a sensor breaks and stops sending data. Can the model guess what it should have said based on the other sensors? XCTFormer was a champion here, filling in the missing data with significantly fewer errors (about 20% better) than the second-best model.
Anomaly Detection (Spotting the Weird Stuff): Can the model spot when a machine is acting strangely? XCTFormer was very good at this, successfully identifying unusual patterns in server data and water treatment systems.

The Bottom Line

The paper claims that the reason previous models failed to use the "group conversation" of data was that they were listening too indirectly. By building a model that listens directly to every single data point's relationship with every other point (using the CRAB engine) and compressing that listening process to keep it fast (using DeCoP), they achieved state-of-the-art results.

They proved that if you build the "choir" correctly—giving it the right tools to understand both positive and negative relationships—it can outperform the "solo singers" after all.

Technical Summary: XCTFormer

Problem Statement

Multivariate time-series analysis is critical for tasks such as forecasting, imputation, and anomaly detection. While real-world variables often share latent dependencies across time and channels, recent findings (e.g., Han et al., 2024) have challenged the assumption that explicitly modeling these cross-channel relationships improves performance. Instead, channel-independent (CI) models, which treat variables separately, often outperform channel-dependent (CD) models. The authors attribute this to a trade-off between model capacity and robustness: CD models may fail to fully exploit their potential due to limitations in how dependencies are captured, often relying on indirect modeling strategies that overlook important interactions or fail to adapt to time-varying dependencies.

Methodology

To address these limitations, the authors propose XCTFormer, a Transformer-based, channel-dependent framework designed to explicitly and directly model pairwise dependencies between every token across both time and channels. The architecture consists of three primary components:

Data Processing Unit:
- Input channels are independently patched to capture local temporal patterns.
- Patches are tokenized and then flattened across both the patch and channel dimensions. This creates a unified sequence where the Transformer can attend to all pairwise relationships (self-lag, cross-channel synchronous, cross-channel lagged, and forward-in-time) in a single step.
Cross-Relational Attention Block (CRAB):
- CRAB modifies the standard Transformer attention mechanism to enhance expressivity and robustness.
- Learnable Non-Boolean Mask: Instead of binary masking, CRAB employs a learnable, real-valued mask applied element-wise to the attention scores. This allows the model to weight dependencies based on learned importance and control the sign of the attention weights.
- Signed Attention Activation (AbsAct): Replacing the standard Softmax, CRAB uses a row-wise normalization based on the absolute sum of values. This allows for negative attention weights, increasing the model's ability to capture a wider range of relationships, while maintaining the bounded Frobenius norm required for training stability.
Dependency Compression Plugin (DeCoP):
- To address the quadratic memory and computational complexity ( $O(N^2)$ ) of modeling all pairwise tokens in high-dimensional datasets, DeCoP is introduced as an optional module.
- It compresses the attention matrix using a learnable transformation, reducing the complexity to linear ( $O(N)$ ) with respect to the sequence length while preserving essential dependency structures. When DeCoP is active, the learnable mask from CRAB is omitted as the direct pairwise structure is no longer fully materialized.

Key Contributions

Identification of a Modeling Gap: The paper highlights that current CD methods often model cross-channel and cross-time dependencies indirectly, failing to realize the full potential of multivariate data.
XCTFormer Framework: A general-purpose architecture that models all pairwise dependencies directly via a token-to-token approach. It integrates:
- CRAB: Enhances expressiveness through learnable importance-aware masks and signed attention activations.
- DeCoP: Mitigates scalability bottlenecks on high-dimensional data through learnable compression.
Comprehensive Evaluation: The model is evaluated on three core tasks (forecasting, anomaly detection, imputation) across multiple benchmarks, demonstrating consistent improvements or competitive performance against strong baselines.

Experimental Results

The authors evaluated XCTFormer on standard benchmarks including ETT subsets, Weather, Electricity, Traffic, and various anomaly detection datasets (SMD, SWaT, PSM, MSL, SMAP).

Imputation: XCTFormer achieved state-of-the-art (SoTA) performance, outperforming the second-best method by an average of 20.8% in MSE and 15.3% in MAE across all datasets.
Forecasting: The model achieved second-best performance on 8 out of 10 evaluation metrics across diverse benchmarks. On a synthetic dataset designed to test robustness to spurious correlations (where the target depends on lagged cross-variate signals), XCTFormer ranked second in accuracy while using significantly fewer parameters and FLOPs than the top-ranked baseline.
Anomaly Detection: The model achieved competitive F1 scores, suggesting effective capture of cross-channel dependencies for identifying unusual patterns.
Ablation Studies: Removing components (learnable mask, signed activation, or the integrated cross-time/channel modeling) consistently degraded performance, validating the necessity of the proposed architectural choices.

Significance and Claims

The paper claims that XCTFormer represents a substantial step toward realizing the full modeling potential of multivariate time-series data by shifting from indirect to direct, token-to-token dependency modeling.

Realizing Potential: The work argues that the underperformance of previous CD models was not due to the inherent difficulty of modeling cross-channel dependencies, but rather the inefficiency of existing indirect strategies.
Robustness: Through synthetic experiments, the authors demonstrate that their approach can distinguish informative cross-variate signals from distractors, showing robustness to spurious correlations.
Limitations: The authors modestly acknowledge that the unified token-to-token attention scales quadratically with the number of tokens, necessitating DeCoP for large datasets. They also note that performance gains are not uniform across all datasets, suggesting the method's effectiveness is sensitive to the underlying dependency structure of the data.

In conclusion, the paper posits that by explicitly capturing the full structure of cross-channel and cross-time interactions through a unified attention mechanism, XCTFormer overcomes the capacity-robustness trade-off that has recently favored channel-independent approaches.