🤖 machine learning

LeNEPA: No-Augmentation Next-Latent Prediction for Time-Series Representation Learning

This paper introduces LeNEPA, a no-augmentation next-latent prediction architecture that achieves robust time-series self-supervised learning across diverse domains without requiring domain-specific tuning, outperforming fixed-recipe baselines in both early convergence and cross-dataset generalization.

Original authors: Alexander Chemeris, Ming Jin, Randall Balestriero

Published 2026-07-02

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Alexander Chemeris, Ming Jin, Randall Balestriero

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Recipe" Trap

Imagine you are a chef trying to teach a robot how to cook.

The Old Way (JEPA): You give the robot a specific recipe for making soup. To help it learn, you tell it, "If the soup looks too salty, add water. If it's too cold, heat it up." These are "augmentations" (changes to the data). The robot learns to recognize soup very well.
The Problem: Now, you want the robot to cook cake. You try to use the exact same "soup recipe" without changing a single instruction. The robot gets confused. It tries to add water to the cake batter or heat it up too early. The recipe fails because it was too specific to soup.

In the world of AI and time-series data (like heartbeats, stock prices, or server temperatures), most current methods rely on these specific "recipes" (augmentations). They work great on the data they were designed for, but if you try to reuse them on a different type of data without tweaking the recipe, they often break.

The Solution: LeNEPA (The "Universal Learner")

The authors propose a new method called LeNEPA. Instead of teaching the robot by messing with the ingredients (augmentations), they teach it a different game: Prediction.

Think of it like a "Next-Word" game, but for time.

The Game: The robot looks at a sequence of events (like a heartbeat: thump-thump... thump-thump...) and is asked to guess what comes next.
The Twist: It does this without any "augmentations." It doesn't try to distort the data to make it harder. It just looks at the raw pattern and predicts the future step.
The Secret Sauce: To keep the robot from getting lazy or confused (a problem called "collapse"), the authors use a special "balance check" called SIGReg. Imagine a tightrope walker who constantly checks their balance to ensure they don't fall to one side. This keeps the robot's understanding of the data stable and useful.

The Experiment: The "Stress Test"

The authors didn't just claim their method was good; they put it through a "stress test" to see if the recipe could travel.

The Setup: They took two methods:
- Method A (JEPA): A strong method tuned specifically for ECG (heart) data.
- Method B (LeNEPA): Their new "no-augmentation" method.
The Test: They trained both methods on heart data (PTB-XL). Then, they took the exact same settings (the same recipe) and tried to train them on Synthetic Diagnostic data (a completely different type of signal, like a made-up medical test).
The Result:
- Method A (JEPA): It was a master chef at heart data, but when forced to cook the synthetic data with the same heart-recipe, it struggled. The performance dropped.
- Method B (LeNEPA): It performed well on the heart data, and when it moved to the synthetic data with the same recipe, it kept performing well. It didn't need to be re-tuned.

The Analogy: Method A was a specialist who only knew how to fix cars. Method B was a mechanic who understood the principles of engines. When you handed them a motorcycle, the specialist (A) was confused, but the mechanic (B) figured it out immediately because their core logic was more flexible.

The "Speed" Bonus

The paper also noticed something interesting about how fast the robot learns.

Method A took a long time to figure out the basics.
Method B (LeNEPA) learned the useful patterns much faster. It reached 80% of its final skill level in just a few thousand steps, while the other method took twice as long.
Analogy: If learning is like running a race, LeNEPA is the sprinter who gets up to speed immediately, while the other method is the marathon runner who takes a while to find their stride.

The "Real World" Check

Finally, the authors tested their method on a massive collection of 128 different time-series datasets (the UCR archive). They froze the brain of their trained robot and asked it to solve these new problems.

The Result: Even though they didn't use any special "tricks" or hand-crafted augmentations, their robot performed almost as well as the current state-of-the-art champions (like MOMENT and Mantis).
The Catch: This was a single test run, so it's not a guarantee that it will always win, but it proves that a "no-augmentation" approach is a very strong contender.

Summary

Current AI: Needs a custom recipe for every new type of data. If you change the data, you have to rewrite the recipe.
LeNEPA: Uses a "predict the next step" strategy that doesn't need custom recipes. It learns the underlying structure of the data directly.
Why it matters: It makes AI for time-series data more portable. You can train it once, keep the settings the same, and apply it to different types of signals (like switching from heart monitors to server logs) without needing a team of engineers to re-tune everything.

In short: LeNEPA is a "universal translator" for time-series data that learns by predicting the future, skipping the messy business of trying to distort the past.

Technical Summary: LeNEPA – No-Augmentation Next-Latent Prediction for Time-Series Representation Learning

1. Problem Statement

Time-series self-supervised learning (SSL) has advanced rapidly, yet practical deployment remains hindered by a heavy reliance on domain-specific view and augmentation strategies. While methods like Joint Embedding Predictive Architectures (JEPA) have shown success (e.g., ECG_JEPA for PTB-XL), their performance is often fragile when the "recipe" (specifically the augmentation and masking strategy) is transferred to a structurally different signal family without re-tuning.

The paper frames a specific, narrower practical question: How does an SSL recipe behave when its method-specific configuration is reused unchanged after the pretraining signal family changes? Rather than comparing against optimally tuned benchmarks, the authors treat this as a "fixed-recipe stress test" to measure the cost of relying on domain-specific invariances (like ECG morphology) when applied to new data types.

2. Methodology: LeNEPA

The authors introduce LeNEPA (Latent Euclidean Next-Embedding Prediction Architecture), a lean, no-augmentation SSL architecture designed to produce useful frozen sequence embeddings without requiring per-dataset view engineering.

Core Architecture

Backbone: A causal Transformer processes a sequence of patch embeddings generated by strided convolutions (ViT-style).
Objective: Autoregressive next-latent-token prediction. At each time step $t$ , the model predicts the latent token at $t+1$ by minimizing the Mean Squared Error (MSE) in a projected space.
No Augmentation: Unlike JEPA variants that rely on masking or cropping, LeNEPA operates directly on the raw sequence without data augmentation.

Stabilization Mechanisms

Standard Next-Embedding Prediction (NEPA) typically relies on a teacher network updated via Exponential Moving Average (EMA) and stop-gradients to prevent collapse. LeNEPA replaces these with:

SIGReg (Sketched Isotropic Gaussian Regularization): Instead of stop-gradients, LeNEPA uses SIGReg to regularize the token embedding distribution toward an isotropic Gaussian. Crucially, the authors find that temporal SIGReg (regularizing the set of tokens within each sample over the temporal axis) is the most effective stabilizer, preventing collapse along the temporal dimension which global regularization misses.
Guillotine Regularization: The prediction loss is computed in a lightweight projected space (using a projector $h_\psi$ ). This projector is discarded at evaluation time, ensuring the learned representations reside in the backbone, not the projection head.

Loss Function

The total loss combines the prediction loss and the regularization term:
$\mathcal{L} = \lambda_{pred}\mathcal{L}_{pred} + \lambda_T \mathcal{L}_{SIG}^{time}$
Where $\mathcal{L}_{pred}$ is the MSE between the projected prediction and the projected target (shifted by one step), and $\mathcal{L}_{SIG}^{time}$ applies isotropic regularization to the temporal token set.

3. Evaluation Protocol

The study employs two distinct evaluation protocols to separate recipe portability from checkpoint transfer:

Fixed-Recipe Pretraining Reuse (PTB-XL vs. Diag):
- Datasets: PTB-XL (real 12-lead ECG) and Diag (a synthetic diagnostic corpus generated via Aionoscope with diverse signal types).
- Setup: Both LeNEPA and a JEPA baseline are trained independently on both datasets.
- Constraint: The "recipes" are fixed based on PTB-XL tuning. The JEPA recipe uses an ECG-tuned masking schedule; LeNEPA uses its fixed no-augmentation configuration. Neither is re-tuned for Diag.
- Probe: Frozen linear probes are trained on intermediate backbone layers (0–8) to measure representation quality.
Frozen-Encoder Check (CauKer to UCR-128):
- A LeNEPA variant pretrained on the CauKer dataset is frozen and evaluated on the full UCR-128 archive using a Random Forest classifier, following the MantisV2 protocol. This tests if a single no-augmentation encoder can generalize to 128 diverse univariate tasks.

4. Key Results

Fixed-Recipe Reuse (PTB-XL vs. Diag)

JEPA Fragility: The ECG-tuned JEPA recipe performs strongly on PTB-XL (in-domain) but degrades significantly when reused unchanged on the structurally different Diag dataset. This highlights the dependency of JEPA on domain-specific augmentation assumptions.
LeNEPA Robustness: LeNEPA preserves useful frozen-probe gains on both datasets without modification. It outperforms the fixed JEPA recipe on Diag in both final performance and training dynamics.
Learning Dynamics: LeNEPA demonstrates faster early representation acquisition. It reaches 80% of its final AUROC/AUPRC gain after 2–5k updates, compared to 5–10k updates for the JEPA readout.

Frozen-Encoder Check (UCR-128)

A single-seed, best-checkpoint run of LeNEPA-CauKer achieves 77.65% mean accuracy on UCR-128 using a Random Forest.
This result is within 1.16 points of Mantis (78.81%) and within 0.24 points of MOMENT (77.89%), despite LeNEPA using a causal backbone, no handcrafted augmentations, and no first-difference branches (structural disadvantages relative to Mantis for this specific benchmark).

Ablations and Diagnostics

Layer Sensitivity: Unlike JEPA, which often peaks at the top layers, LeNEPA and NEPA achieve optimal probe scores at intermediate layers (typically $L_4$ or $L_5$ ). The final layer often underperforms, suggesting deeper blocks specialize in the prediction task while intermediate layers retain more general features.
SIGReg Axis: Temporal SIGReg was the only regularization placement that provided sustained gains across both datasets.
Projector: Using a disposable projector (Guillotine style) improves performance over direct backbone loss computation, validating the separation of loss space from representation space.

5. Significance and Claims

The paper concludes that no-augmentation latent prediction is a promising candidate recipe for time-series SSL when the operational constraint is to reuse an SSL recipe with minimal per-dataset view engineering.

Modest Claims: The authors explicitly state they are not claiming a new "foundation model" that outperforms all tuned baselines. Instead, they demonstrate that removing augmentation dependencies reduces the fragility of SSL recipes when signal families change.
Practical Impact: The results suggest that for applications requiring portable pretraining (e.g., industrial telemetry, finance, physiology), a no-augmentation objective like LeNEPA offers a more robust alternative to JEPA variants that require careful, domain-specific view engineering.
Limitations: The study acknowledges that the convolutional tokenizer still requires dataset-specific design (kernel size, stride) and that the synthetic Diag corpus serves as controlled evidence rather than a direct proxy for all real-world diagnostic transfer. The UCR-128 result is based on a single seed and best checkpoint, serving as an existence proof rather than a definitive leaderboard claim.

Source Code: Available at https://github.com/langotime/lenepa-milets-2026.