🔭 astrophysics

Domain-Informed Multi-View Self-Distillation for Astronomical Light-Curve Representation Learning with JEPA

This paper proposes a domain-informed multi-view self-distillation framework using Joint-Embedding Predictive Architecture (JEPA) to learn robust representations of irregular astronomical light curves, demonstrating superior performance over hand-crafted features in classification, few-shot learning, and cross-domain adaptation tasks.

Original authors: Yicheng Rui

Published 2026-06-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yicheng Rui

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to recognize different types of stars by looking at how their brightness changes over time. These brightness records are called light curves.

In the real world, astronomers can't watch the stars 24/7. The sun comes up, clouds roll in, and telescopes have to move around. This means the data they collect is messy: there are huge gaps in time, the measurements are sometimes fuzzy or uncertain, and some stars pulse quickly while others take years to complete a cycle.

Existing AI models, which are great at reading smooth, regular text or stock market data, often get confused by this "messy" star data. They tend to fail where human-made rules (hand-crafted features) still succeed.

This paper introduces a new AI system designed specifically to handle this messiness. Here is how it works, using simple analogies:

1. The "Three-Perspective" Strategy (Multi-View Self-Distillation)

Instead of just showing the AI the raw, messy list of numbers, the researchers teach it to look at the same star data through three different lenses simultaneously. Think of it like trying to identify a person in a crowd:

Lens 1 (The Raw View): Looking at the person as they walk by, with all their natural movements and the noise of the crowd.
Lens 2 (The Frequency View): Looking at a "soundwave" of their movement. This helps the AI spot the rhythm or beat of the star, even if the timing is irregular.
Lens 3 (The Folded View): Imagine taking the star's long, messy timeline and folding it up so that every cycle lines up perfectly on top of each other. This reveals the star's unique "shape" or "dance move."

The AI is trained to realize that all three views describe the same star. By forcing the AI to agree on the identity of the star across these three different perspectives, it learns a much deeper, more robust understanding than if it just looked at one view. This is called self-distillation.

2. Specialized Tools for Messy Data

The researchers didn't just use a standard AI; they built custom tools to handle the specific problems of astronomy:

The "Flexible Clock" (C-RoPE): Standard AI models usually count steps (1, 2, 3...). But star observations happen at random times (1:00 PM, then 3:45 PM, then 10:00 AM the next day). The team gave the AI a "flexible clock" that understands the actual time gaps between observations, not just the order of the list.
The "Confidence Meter" (EANE): Sometimes a telescope is very sure of a measurement; other times, it's just a guess. Standard AI treats every number as a hard fact. This new model treats every number as a "cloud of probability." If the telescope is unsure, the AI knows to be less confident too, rather than getting confused by the noise.

3. The Results: Beating the Experts

The researchers tested their new model on a famous benchmark called StarEmbed, which is like a final exam for star classification.

The Score: The new model beat the old "human-made rules" in 15 out of 16 categories. This is a big deal because, until now, human-made rules were usually the best at this specific task.
Learning from Few Examples: Even when the AI was only shown one single example of a new type of star, it performed better than the human rules. This is like learning a new language after hearing just one sentence.

4. What Else Can It Do?

Beyond just naming stars, the paper shows the AI can:

Find Twins: If you show it one star, it can instantly find other stars that look exactly like it (similarity search).
Guess the Physics: It can estimate the physical properties of the star, like how long its cycle is or how bright it gets (parameter estimation).
Spot Glitches: It can detect if the telescope itself had a calibration error (zero-point drift) that made all the data look slightly wrong.
Work on Other Data: The team tried applying this "three-lens" method to 12 different types of messy, irregular data from other fields (like finance or medicine). It performed as well as or better than the best existing methods on half of them, suggesting this approach might work for other messy data types too.

The Bottom Line

The paper argues that there is no single "magic bullet" AI that works perfectly for every type of data. Instead, to understand the universe (or any complex system), you need to build an AI that respects the specific rules and quirks of that domain. By combining domain knowledge (how stars actually behave) with modern AI techniques (JEPA and self-distillation), they created a system that finally understands the messy, irregular language of the stars better than humans ever could.

Technical Summary: Domain-Informed Multi-View Self-Distillation for Astronomical Light-Curve Representation Learning with JEPA

Problem Statement

Astronomical light curves, which describe the temporal brightness variations of celestial objects, are critical for understanding the dynamic universe. However, learning robust global representations from these time series is challenging due to three distinct characteristics that differ from standard time-series data:

Extremely Uneven Sampling: Ground-based observations are limited to nighttime, creating inevitable gaps between nights and seasons, alongside irregular intra-night cadences.
Complicated Noise: Noise arises from quantum effects, atmospheric turbulence, weather, and instrumentation. Crucially, this noise is temporally correlated, and for faint sources, noise can exceed the signal itself.
Wide Dynamic Range of Timescales: Phenomena span from minutes to years, and single objects may exhibit multiple variation periods.

Existing "off-the-shelf" time-series foundation models (TSFMs), such as Transformer-based architectures, often underperform on light curves. Standard time embeddings fail to capture sampling density, and auto-regressive pre-training (e.g., GPT-style) tends to overfit on high-noise sequences. Consequently, these models frequently fail to outperform hand-crafted features in benchmarks like StarEmbed.

Methodology

The authors propose a Domain-Informed Multi-View Self-Distillation framework based on the Joint-Embedding Predictive Architecture (JEPA). The core innovation lies in combining domain-specific inductive biases with a multi-view learning strategy to handle irregularity and noise.

1. Multi-View Architecture
The model processes three distinct views of each light curve, assuming they share similar semantics derived from the raw data:

Raw Sequence ( $T$ ): The original time-series data $\{(t_i, m_{t_i}, e_{t_i})\}$ .
Generalized Lomb-Scargle (GLS) Periodogram ( $P$ ): A power spectrum used to extract periodicity and diagnose targets with multiple oscillation modes.
Phase-Folded Light Curve ( $T_p$ ): The raw data folded over the dominant period ( $P_o$ ) to extract the morphology of periodic variations.

2. Specialized Encoders
Each view is processed by a separate Transformer encoder equipped with two novel components:

Continuous Rotary Positional Embedding (C-RoPE): Unlike discrete positional embeddings, C-RoPE maps continuous time coordinates $t$ to complex rotations in the hidden space. This ensures the attention mechanism is invariant to absolute time shifts and depends only on relative intervals ( $\Delta t$ ), accommodating arbitrary temporal spacing.
Error-Aware Numeric Embedding (EANE): Instead of treating measurements as point estimates, EANE models them as probability distributions $N(m_{t_i}, e_{t_i}^2)$ . It computes the expected embedding vector by integrating over the Gaussian PDF of the measurement uncertainty, propagating error information through the network.

3. Training Objective: LeJEPA
The model is trained using LeJEPA regularization, a multi-view self-distillation loss. The objective consists of two terms:

Alignment Loss ( $L_{inv}$ ): Minimizes the distance between the latent embeddings of the three views and their centroid, ensuring semantic consistency across views.
Regularization Loss ( $L_{SIGReg}$ ): Aligns the empirical characteristic function of the projected features with a standard Gaussian distribution. This optimizes the latent space for linear probing and radial-based k-NN.

Key Contributions

First Deep Learning Model to Surpass Hand-Crafted Features: The proposed model is the first foundation model to outperform hand-crafted features (based on FATS) on the StarEmbed classification benchmark across 15 of 16 metrics.
Novel Embedding Mechanisms: Introduction of C-RoPE for irregular sampling and EANE for measurement uncertainty, addressing specific astronomical data characteristics that generic models ignore.
Domain-Informed Multi-View Strategy: Demonstrating that combining raw, spectral (GLS), and morphological (phase-folded) views via self-distillation yields superior representations compared to single-view approaches.
Cross-Domain Adaptability: The framework shows potential beyond astronomy, achieving state-of-the-art (SOTA) or competitive results on 5 of 12 irregular time-series datasets in the PYRREGULAR benchmark after domain adaptation.

Experimental Results

StarEmbed Benchmark: On the classification task, the full model achieved superior performance in 15/16 metrics compared to hand-crafted features and other baselines (Chronos-2, Astromer-2, FALCO).
Few-Shot Learning: In linear probing with limited data, the model consistently outperformed hand-crafted features. With 1 sample per class, it achieved a macro-F1 of 42.56 ± 7.21, compared to 30.28 for hand-crafted features. With 100 samples per class, it reached 63.58 ± 1.20.
Ablation Studies:
- Removing C-RoPE or EANE degraded performance, confirming the necessity of handling irregular time and uncertainty.
- Removing any of the three views (Raw, GLS, Phase-Folded) reduced accuracy, validating the multi-view approach.
- Using standard CLIP loss instead of LeJEPA resulted in lower performance.
Applications:
- Similarity Search: Successfully retrieved light curves of the same class (e.g., RRc variables) based on L2 distance in the embedding space.
- Parameter Estimation: Linear probes on embeddings successfully recovered period and amplitude of sinusoidal signals ( $R^2 \approx 0.81$ and $0.99$), though phase was not recovered due to the model's time-translation invariance.
- Drift Detection: The model effectively detected photometric zero-point drifts, with EANE and multi-view inputs improving detection of small drifts.

Significance and Claims

The paper argues that successful time-series representation learning requires domain-specific inductive biases rather than a universally optimal architecture. The authors claim that "off-the-shelf" models fail because they lack the specific priors needed for astronomical data (e.g., time-translation invariance, uncertainty propagation, and periodicity extraction).

By integrating domain knowledge (GLS, phase-folding) directly into the architecture and training objective, the proposed method bridges the gap between hand-crafted feature engineering and deep learning. The results suggest that while the specific views (GLS/Phase-Folding) are hard to replace with learnable modules for light curves due to the non-differentiable nature of Nyquist frequencies in uneven sampling, the multi-view self-distillation framework is a transferable strategy. The authors conclude that for other domains, similar success requires tailoring views and encoders to the specific underlying dynamics of that domain's time series.