Technical Summary: Falcon-X

Problem Statement

Time Series Foundation Models (TSFMs) are reshaping forecasting by leveraging large-scale cross-domain pretraining. However, existing TSFMs face two fundamental limitations when handling heterogeneous multivariate data:

1. Semantic Misalignment: Most models operate directly in the raw variate space. When mixing heterogeneous physical quantities (e.g., temperature vs. traffic volume) from different datasets, standard group attention mechanisms fail to align their distinct semantics. This leads to "semantic collapse," where the model learns spurious, dataset-specific correlations rather than transferable temporal structures.
2. Limited Relational Expressivity: Real-world systems exhibit complex interactions, including both synergistic (positive) and antagonistic (negative) dependencies. Standard attention mechanisms, relying on non-negative softmax scores, primarily capture aggregative effects and struggle to explicitly model opposing dynamics or antagonistic relationships.

Current approaches often flatten multivariate series or use group attention within the raw space, which either scales poorly with dimensionality or fails to disentangle heterogeneous semantics, limiting zero-shot transfer capabilities.

Methodology: Falcon-X

Falcon-X is an encoder-only foundation model (591M parameters) designed to decouple physical variates from the interaction space. It introduces a Unified Latent Prototype Space paradigm, consisting of four main stages:

1. Pre-processing and Tokenization

• Normalization: Inputs undergo an arcsine transformation using instance-wise mean and standard deviation computed only from observed values to preserve missing entries.
• Tokenization: The sequence is augmented with relative timestamps and a binary observation mask. It is then partitioned into non-overlapping patches and projected into a hidden dimension via a residual patch embedding mechanism.

2. Time Attention

Falcon-X employs an encoder-only Transformer architecture to first capture intrinsic temporal patterns. $n$ layers of Time Attention are applied independently to each variate along the time dimension. This ensures that robust temporal dynamics are distilled before any cross-variate interaction occurs, preventing premature semantic entanglement.

3. Variate Attention (Core Innovation)

This stage bridges the gap between heterogeneous inputs and a unified representation:

• Unified Prototype Diff-Attention (UPDA): Instead of mixing raw variates, Falcon-X projects them into a fixed-dimensional latent prototype space ($C$). It utilizes a Differential Attention mechanism with two learnable key matrices: $K_{pos}$ (synergistic) and $K_{neg}$ (antagonistic).
  • The model computes attention scores against both keys: $A_{pos} = \text{softmax}(QK_{pos}^T)$ and $A_{neg} = \text{softmax}(QK_{neg}^T)$.
  • The unified representation is derived by aggregating values based on the differential score: $H_C = (A_{pos} - \lambda A_{neg})^T V$.
  • An orthogonality loss constrains $K_{pos}$ and $K_{neg}$ to ensure semantic distinctiveness. This allows the model to explicitly capture both positive and negative affinities.
• Latent Entity Attention (LEA): Once aligned in the shared prototype space, global cross-variate interactions are performed via standard Multi-Head Attention. Because all entities reside in the same dimension-agnostic semantic space, this facilitates seamless zero-shot transfer across datasets with different variate counts and physical meanings.
• Variate Reassembly Router (VRR): To reconstruct specific trajectories, a request-and-dispatch mechanism is used. A routing request ($R_{req}$) from the original temporal embedding is matched against a prototype index ($P_{idx}$) to retrieve relevant semantic payloads from the source context ($S_{ctx}$).
  • The final output fuses the original temporal embeddings with the reassembled cross-variate representations via a gated residual connection: $\hat{H} = H_T + G(H_T) \odot H_V$. This gate dynamically regulates the fusion, preventing semantic interference in weakly correlated systems while leveraging dependencies in strongly correlated ones.

4. Forecasting Head

The model adopts a probabilistic forecasting paradigm, predicting future distributions across a set of quantiles using a Quantile Loss.

Key Contributions

1. A Novel Heterogeneous Modeling Paradigm: Falcon-X shifts from raw-space mixing to a unified latent prototype space. This creates a universal coordinate system that resolves semantic discrepancies across datasets, enabling principled zero-shot knowledge transfer.
2. Architectural Innovation:
   • Differential Prototype Attention: Explicitly models both synergistic and antagonistic dynamics, overcoming the limitations of non-negative attention.
   • Variate Reassembly Router: A gated mechanism that adaptively fuses global context with local temporal signals, ensuring robustness across diverse dependency strengths.
3. Empirical Excellence: The model demonstrates state-of-the-art performance on comprehensive benchmarks, validating its structural adaptability.

Experimental Results

Falcon-X was evaluated on GIFT-Eval and fev-bench, two large-scale benchmarks covering diverse domains (nature, energy, finance, healthcare, etc.).

• GIFT-Eval: Falcon-X achieved the best overall performance with 0.666 MASE and 0.453 CRPS. It outperformed strong baselines including STRIDE (+1.2% MASE), Toto-2.0-FT (+1.9% MASE), and Timer-S1 (+3.9% MASE). Notably, its advantage increased with longer forecasting horizons, achieving the best medium-term (0.68 MASE) and long-term (0.70 MASE) results.
• fev-bench: Falcon-X ranked closely behind Chronos-2 (0.652 vs. 0.645 MASE) while relying strictly on endogenous target series, outperforming other recent models like TiRex and Moirai 2.0.
• Ablation Studies: Removing the negative prototype key ($K_{neg}$) caused the most severe performance drop, confirming the necessity of modeling antagonistic relationships. Joint training (univariate + multivariate) proved superior to two-stage curriculum learning.
• Scaling: The model adheres to neural scaling laws, showing consistent performance improvements as parameters scale from 59M to 591M.

Significance and Claims

The paper claims that Falcon-X offers a principled and scalable paradigm for complex multivariate environments. By decoupling physical variates from the interaction space and explicitly modeling dual dependencies (synergistic and antagonistic), it addresses the semantic alignment and relational expressivity bottlenecks of current TSFMs. The authors posit that this approach enables robust zero-shot structural transfer, allowing the model to learn reusable cross-domain patterns without being coupled to raw variate dimensions. The work is presented as a step toward more unified and expressive foundation models for time series, with the code publicly released to support future research.