🤖 machine learning

Faithful Embeddings of Irregular and Asynchronous Data for Online Log-NCDEs

This paper proposes a continuous and injective embedding method for Log-NCDEs that directly constructs log-signatures from data increments without interpolation, enabling accurate, efficient, and robust online modeling of irregular and asynchronous time-series data.

Original authors: Benjamin Walker, Alexandre Bloch, Lingyi Yang, Sam Morley, Terry Lyons

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Benjamin Walker, Alexandre Bloch, Lingyi Yang, Sam Morley, Terry Lyons

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: Trying to Fill in the Blanks

Imagine you are trying to understand a story, but the pages you have are torn, missing, and arrived at random times. Sometimes you get a whole page; other times, you get just a single sentence.

Most computer models (like the ones that power AI chatbots or predict stock prices) are built to read stories where the pages arrive perfectly in order, one after another, with no gaps. To make messy, irregular data fit these models, scientists usually try to "fill in the blanks." They guess what happened between the pages they have.

Interpolation: Drawing a smooth line between two dots.
Imputation: Guessing a missing number based on the ones around it.

The Paper's Critique: The authors argue that this "guessing" is dangerous. If you draw a smooth line between two data points, you might invent a story that never happened. You are feeding the computer fake information, which can lead to bad decisions.

The Solution: The "Faithful" Messenger

Instead of guessing what happened in the gaps, the authors propose a new way to send the data to the computer. They call this a "Faithful Embedding."

Think of it like a messenger service that only reports exactly what it saw, nothing more.

Old Way (Interpolation): The messenger says, "I saw you at 1:00, and I saw you at 2:00, so I assume you walked in a straight line between them." (This might be wrong; maybe you ran, stopped, or went to a store).
New Way (Faithful): The messenger says, "I saw you at 1:00. Then, at 2:00, I saw you again. Here is the difference in your location between those two times."

The computer doesn't need to know how you moved in between; it only needs to know the start, the end, and the change. This keeps the data "faithful" to reality.

How It Works: The "Log-Signature" Recipe

The paper introduces a specific mathematical tool called a Log-NCDE (Neural Controlled Differential Equation). To make this work with messy data, they created a special recipe for turning observations into math.

Imagine you are tracking a hiker's journey through a forest, but you only get text messages from them at random times.

The Increment: Instead of asking "Where are you?", the system asks "How far did you move since your last message?" and "Which direction?"
The Count: It also notes, "Hey, a message just arrived!" This is crucial because if the hiker stays still but sends a message saying "I'm still here," that's different from them not sending a message at all. The system counts these events to ensure it doesn't lose track.
The "Log-Signature": This is the magic sauce. It's a way of summarizing a whole chunk of the journey (an interval) into a single, compact "fingerprint."
- Instead of looking at every single step the hiker took, the system looks at the total change over a specific time block.
- It captures not just the distance, but the twists and turns (like if the hiker walked in a circle). This is called capturing "higher-order information."

Why This is a Game-Changer

The authors prove two main things:

No Guessing Needed: You don't need to invent a smooth path between data points. If you just record the changes and the time gaps accurately, a smart computer model can learn just as well (or better) than if you had guessed the path.
Speed and Efficiency: Because the system summarizes chunks of time into these "fingerprints" (log-signatures), it can process data much faster.
- Analogy: Imagine reading a 1,000-page book.
  - Old Way: You read every single word, one by one, even the ones you already know.
  - New Way: You summarize every chapter into a single sentence, then read those sentences. You get the whole story much faster, and you can do it while the book is still being written (online computation).

The Results: Tougher Data, Better Answers

The team tested this on two types of challenges:

Synthetic Data (The "Fake" World): They created computer simulations where data arrived irregularly, asynchronously (different channels at different times), and sparsely (lots of missing info).
- Result: Their method was robust. Even when 95% of the data was missing, their model still worked well. Other models that relied on "filling in the blanks" failed miserably because their guesses were wrong.
Real World Data (The "Real" World): They tested on standard datasets used for classifying things like heartbeats or worm movements.
- Result: Their models were faster (up to 3,000 times faster in some cases) and more accurate, especially when the data was messy or sparse.

The Takeaway

This paper argues that when dealing with messy, real-world data, honesty is the best policy. Don't try to guess what happened in the gaps. Instead, record exactly what you saw, how much it changed, and when it happened. By using a special mathematical summary (the log-signature) to package this honest data, you can build AI models that are faster, more accurate, and much better at handling the chaos of the real world.

Technical Summary: Faithful Embeddings of Irregular and Asynchronous Data for Online Log-NCDEs

Problem Statement

Continuous-time machine learning models, such as Neural Controlled Differential Equations (NCDEs) and Log-NCDEs, offer a natural framework for processing irregular, asynchronous, and sparse time-series data. However, a critical design bottleneck remains: how to embed discrete observations into a continuous-time path without introducing artifacts.

Standard approaches rely on interpolation or imputation to reconstruct a continuous observation path before feeding it into the model. As illustrated in the paper, these methods suffer from significant drawbacks:

Imputation fills missing values with data never observed in practice.
Interpolation can generate values outside the observed range of the data or impose artificial dynamics between measurements.
These reconstruction steps introduce "unobserved behaviour" into the representation, forcing the model to consume potentially unrealistic inputs.

The paper argues that a faithful continuous-time representation should preserve only the information that was actually observed, leaving the reconstruction of unobserved behavior to the modeling process itself rather than the embedding process.

Methodology

1. Theoretical Foundation: Faithful Embeddings

The authors formalize the concept of a faithful embedding. They prove a universality transfer result (Lemma 1) stating that if the data space $X$ and the model input space $P$ are Hausdorff topological spaces, and the embedding $\phi: X \to P$ is continuous and injective, then the universality of a downstream model on compact subsets of $P$ transfers to the original data space $X$ .

This implies that for universal models like NCDEs and Log-NCDEs, the embedding need not reconstruct the path; it merely needs to be a continuous, injective map that preserves the distinguishability of distinct data streams.

2. Proposed Embedding: Log-NCDE Control Path

Building on the rectilinear control path introduced for NCDEs by Morrill et al. [2022], the authors construct a faithful embedding specifically for Log-NCDEs. Unlike standard NCDEs which use pointwise values of a control path, Log-NCDEs operate on interval log-signatures.

The proposed embedding operates as follows:

Input Representation: The input consists of a stream of discrete observations $x = ((t_0, s_0, x_0), \dots, (t_n, s_n, x_n))$ and continuously observed variables $C$ (e.g., time).
Incremental Encoding: Instead of interpolating values, each observation is converted into a Lie-algebra increment $\Delta_i$ $Δ_{i}$ . This increment captures:
- The change in value for each observed channel since its last observation.
- An increment in a dedicated "count coordinate" to distinguish between a repeated value and the absence of a new observation.
- Optionally, higher-order local information (e.g., precomputed signed areas) if available.
Interval Composition: For any query interval $[\alpha, \beta)$ $[α, β)$ , the embedding constructs a signature factor $G_{\alpha, \beta}$ $G_{α, β}$ by composing:
- Signature factors from the continuously observed variables (e.g., time) over the gaps between events.
- Exponential factors $E_i = \exp(\Delta_i)$ for each discrete event occurring within the interval.
Log-Signature Extraction: The interval log-signature $\Phi_{\alpha, \beta}$ is obtained by applying the logarithm to the composed signature $G_{\alpha, \beta}$ .

This construction allows for online computation and parallel-in-time processing via associative scans, as the log-signature for an interval depends only on observations up to the end of that interval.

3. Model Architecture: Log-SLiCE

The authors implement this embedding within Structured Linear CDEs (SLiCEs), specifically the Log-SLiCE variant.

The hidden state evolution on an interval is governed by a Log-ODE update driven by the interval log-signature.
For linear models, the interval flow has a closed-form solution involving matrix exponentials of the log-signature.
This structure allows the hidden states across intervals to be computed in parallel using an associative scan, significantly reducing computational complexity compared to sequential ODE solvers.

Key Contributions

Faithful Embedding Formalization: The paper proves that continuous, injective embeddings transfer universality from the model input space back to the data space, justifying the removal of interpolation steps.
Novel Embedding Construction: A continuous and injective embedding for irregular streams that records observations as tensor-algebra factors (increments) and composes them into interval log-signatures without interpolation.
Efficient Log-SLiCE Implementation: A parallel-in-time algorithm for Log-NCDEs that leverages the interval log-signature formulation, enabling efficient training and inference on long sequences.
Empirical Validation: Comprehensive experiments demonstrating accuracy, efficiency, and robustness to irregularity and sparsity.

Experimental Results

1. Efficiency and Accuracy (EigenWorms)

On the EigenWorms dataset (17,984 observations per series), the Log-ODE method combined with Log-SLiCE (BD-SLiCE) achieved:

Accuracy: 86.1% test accuracy (vs. 75.0% for standard NCDE), an 11.1 percentage point improvement.
Speed: Reduced training step time by a factor of nearly 3,000 (from 26.02s to 0.009s).
Memory: Maintained comparable GPU memory usage to standard NCDEs.

2. Generalization to Irregular Sampling (Synthetic Coupled Data)

Using a synthetic sinusoidal system, the authors tested cross-regime generalization.

Models trained on asynchronous sparse data (the hardest regime) remained robust across all other sampling regimes (synchronous regular, synchronous irregular, asynchronous irregular).
Models trained on regular data failed to generalize to asynchronous sparse settings, suggesting they learned shortcuts based on fixed sampling grids.
The faithful embedding allowed models to learn the underlying dynamics regardless of the observation grid.

3. Exploitation of Higher-Order Information (Brownian Motion)

In a linear system driven by Brownian motion where the solution depends on second-order Lie bracket terms (Lévy area):

Models utilizing Level 2 log-signatures (including area information) achieved significantly lower Mean Squared Error (MSE) than those using only Level 1 (increments).
The approach successfully incorporated precomputed higher-order information, demonstrating flexibility beyond first-order increments.

4. Robustness to Input Dropping (UEA Benchmarks)

On six UEA multivariate time-series classification datasets, the authors simulated severe input dropping (up to 95% of observations removed).

BD-SLiCE with the proposed embedding maintained high performance, dropping only from 64.0% (no drop) to 62.5% (95% drop).
It achieved the best average rank across all dropping levels, outperforming recurrent baselines (S5, LRU, S6, Mamba) and standard NCDEs, particularly under severe sparsification.

Significance and Claims

The paper claims that continuous-time representations should prioritize preserving observed data over reconstructing unobserved behavior. By formalizing "faithful embeddings" and demonstrating that they preserve the universality of downstream models, the authors provide a theoretical justification for bypassing interpolation.

The proposed method offers a faithful, online, and efficient alternative to interpolation-based embeddings. It decouples input sampling times from output query grids, supports parallel computation, and demonstrates superior robustness to irregularity, asynchrony, and data sparsity. The results suggest that for universal continuous-time models, the quality of the embedding lies in its ability to faithfully encode observed increments and intervals, rather than in its ability to guess the path between observations.