Astromer 2

This paper introduces Astromer 2, an enhanced foundational model for light curve analysis that leverages self-supervised pre-training on 1.5 million MACHO survey observations and weighted per-sample embeddings to significantly outperform its predecessor and prior models in classification tasks, particularly when trained with limited labeled data.

The Gist

Imagine the night sky as a giant, bustling city where every star is a person with a unique daily routine. Some stars blink like a lighthouse, others pulse like a heartbeat, and some dim and brighten because they are being hugged by a partner star. Astronomers call these changing brightness patterns light curves.

For a long time, to understand these stars, astronomers had to act like detectives, manually measuring every wiggle and wiggle in the data. It was slow, tedious, and often missed the bigger picture.

Then came Astromer, an AI designed to be a "super-detective." The original version (Astromer 1) was great, but the team just released Astromer 2, a much smarter, more experienced version. Here is how it works, explained simply:

1. The "Silent Library" Training (Self-Supervised Learning)

Imagine you want to teach a child to recognize different types of music. You could give them a list of songs with labels like "Jazz" or "Rock," but that's expensive and takes forever to write.

Instead, Astromer 2 uses a trick called Self-Supervised Learning.

• The Analogy: Imagine you have a library of 1.5 million light curves (the "unlabeled" data). You take a book, cover up random paragraphs, and ask the AI: "Based on the sentences before and after, what words should be here?"
• The Result: The AI has to study the patterns of the stars so deeply that it can guess the missing parts perfectly. It doesn't need a teacher telling it "this is a Cepheid star"; it just learns the shape of the story. This is how it builds a massive "brain" of knowledge without needing expensive labels.

2. The Upgrade: From "Sketch" to "Masterpiece"

The original Astromer was like a sketch artist who could draw a decent face. Astromer 2 is like a master painter with a much larger canvas.

• More Layers: The new model has six "layers" of thinking (up from two). Think of this as having six different experts looking at the same star, each focusing on a different detail.
• The "Mask" Token: In the old version, when the AI had to guess a missing number, it just saw a blank space. In Astromer 2, they replaced the blank space with a special "I'm missing something" flag. This helps the AI realize, "Ah, this spot is hidden, I need to work harder to figure it out!"
• Weighted Wisdom: The new model doesn't just look at the final answer. It listens to the "whispers" of all the intermediate experts (the middle layers) to make its final decision. It's like a committee where the middle managers often have the best insights, not just the CEO.

3. The "Few-Shot" Superpower

The real magic happens when the AI meets a new type of star it has never seen before, and you only have 20 examples to teach it.

• The Old Way: A traditional AI would be confused. "I've only seen 20 examples? I can't learn anything!"
• The Astromer 2 Way: Because it already studied 1.5 million stars in the "Silent Library," it already knows the grammar of the universe. When you show it 20 new examples, it's like showing a fluent speaker of Spanish a few words of Italian; they can guess the meaning almost immediately.
• The Result: Astromer 2 improved its accuracy by 15% on new data (the ATLAS survey) compared to older models, even when it had almost no labeled data to work with.

4. Why This Matters

Think of the universe as a massive, unorganized attic full of boxes.

• Before: Astronomers had to open every box, read the contents, and sort them by hand.
• Now: Astromer 2 is a robot that can look at the shape of the boxes and instantly say, "This one is a variable star, that one is a binary system, and that one is just noise."

It allows astronomers to:

1. Find the rare gems: Spot unusual stars that don't fit the standard molds.
2. Save time: Process millions of stars in the time it used to take to process a few hundred.
3. Look further: Because it works so well with very little data, it can be used on new telescopes and surveys that haven't even finished collecting data yet.

The Bottom Line

Astromer 2 is like giving astronomers a pair of glasses that don't just make things look clearer, but actually understand what they are seeing. By teaching the AI to play "fill-in-the-blanks" with millions of stars first, it becomes a master of the cosmic language, ready to classify the universe's secrets with just a few hints.

Technical Summary

Here is a detailed technical summary of the paper "Generalizing Across Astronomical Surveys: Few-Shot Light Curve Classification with Astromer 2."

1. Problem Statement

Astronomical light curve analysis is essential for characterizing stellar objects, but it faces significant challenges:

• Data Scarcity: High-quality labeled data for specific variable star classes is often scarce and expensive to obtain.
• Irregular Sampling: Light curves have irregular time intervals and varying lengths, making them difficult to process with standard deep learning architectures designed for fixed sequences.
• Domain Shift: Models trained on one survey (e.g., MACHO) often struggle to generalize to others with different observational cadences and noise characteristics (e.g., ATLAS).
• Feature Engineering: Traditional methods rely on manually engineered features, which may introduce bias or fail to capture complex, non-linear patterns in the data.

The paper addresses these issues by proposing Astromer 2, an enhanced foundational model designed to learn robust, general-purpose embeddings from unlabeled light curves via self-supervised learning, enabling high-performance classification even with very few labeled samples (Few-Shot Learning).

2. Methodology

A. Model Architecture (Astromer 2)

Astromer 2 is an evolution of the original Astromer (Astromer 1), based on a Transformer encoder architecture adapted for time-series data.

• Input Embedding: Unlike NLP models with discrete tokens, Astromer processes continuous magnitude values paired with observation times (MJD). It uses a positional encoder (PE) adapted for irregular sampling and linearly projects magnitudes into vector space.
• Masking Strategy: The model employs a self-supervised masked modeling task.
  • Astromer 1: Masked magnitudes were simply removed from the attention calculation.
  • Astromer 2 Improvement: Masked magnitudes are replaced by a trainable, zero-initialized MASK token. This allows the model to explicitly recognize masked positions without leaking information from the similarity matrix.
• Encoder Depth: The encoder depth was increased from 2 blocks (Astromer 1) to 6 blocks, each with 4 attention heads. This significantly increases the parameter count from ~0.66M to ~3.95M, enhancing representational capacity.
• Regularization: Dropout layers were added after self-attention calculations to prevent overfitting.

B. Pretraining Task

• Objective: Predict the magnitudes of a "probed" subset (50% of observations) within a light curve window.
• Loss Function: Root Mean Square Error (RMSE).
  • Astromer 2 Improvement: The loss is weighted by observational uncertainties. Uncertainties are normalized, and their reciprocals serve as weights. This ensures the model prioritizes high-confidence observations during reconstruction.
• Data: Pretrained on 1.5 million single-band light curves from the MACHO survey (bands B and R).

C. Downstream Classification (Few-Shot Learning)

• Fine-tuning: The pretrained encoder is fine-tuned on small labeled datasets (20, 100, or 500 samples per class) using the same masked reconstruction task.
• Embedding Aggregation: Instead of using only the final output token, Astromer 2 generates weighted per-sample embeddings. It averages the attention vectors from all 6 encoder blocks, weighted by learnable parameters ($\gamma$). This allows the classifier to leverage intermediate representations.
• Classifier: A Multi-Layer Perceptron (MLP) with three hidden layers (1024, 512, 256 units) and ReLU activation is trained on top of the frozen (or fine-tuned) embeddings to predict class labels.

3. Key Contributions

1. Astromer 2 Model: An improved foundational model with a deeper architecture (6 blocks vs. 2), a trainable MASK token, and uncertainty-weighted loss functions.
2. Weighted Per-Sample Embeddings: A novel aggregation strategy that combines intermediate layer outputs, proving that intermediate representations contain critical discriminative information often lost in final-layer-only approaches.
3. Cross-Survey Generalization: Demonstrated that a model pretrained on MACHO data can effectively generalize to the ATLAS survey, despite significant differences in magnitude distributions and observation cadence (MACHO: ~3-4 days; ATLAS: ~15 minutes).
4. Few-Shot Performance: Established a new benchmark for light curve classification, achieving state-of-the-art results with as few as 20 labeled samples per class.

4. Results

• Pretraining Performance: Astromer 2 achieved an RMSE of 0.113 on the reconstruction task, a significant improvement over Astromer 1 (0.148).
• Classification on Alcock (MACHO) Dataset:
  • Astromer 2 consistently outperformed Astromer 1 and the original Donoso-Oliva et al. (2023) results.
  • With 500 samples per class, Astromer 2 achieved an F1 score of 74.2%, compared to 71.0% for Astromer 1.
• Classification on ATLAS Dataset (Out-of-Distribution):
  • The most significant gains were observed here. With only 20 samples per class, Astromer 2 achieved an F1 score of 71.5%, a 15% improvement over previous models (which scored ~55-60%).
  • Astromer 2 with 20 samples outperformed Astromer 1 trained with 500 samples, highlighting superior generalization capabilities.
• Attention Analysis:
  • Visualization of attention weights revealed that the model consistently focuses on points of maximum and minimum brightness (extrema) across different datasets and star types.
  • The learned $\gamma$ weights showed that intermediate blocks (blocks 2-5) contribute more to the final classification than the initial or final blocks, validating the weighted embedding strategy.
• t-SNE Visualization: Embeddings from intermediate and deeper layers showed clear separation of classes (e.g., Cepheids, RR Lyrae, Eclipsing Binaries) even without label information during pretraining.

5. Significance and Conclusion

• Efficiency: Astromer 2 demonstrates that self-supervised learning on massive unlabeled datasets can produce embeddings that are highly effective for downstream tasks with minimal labeled data. This is crucial for future large-scale surveys (e.g., LSST/Rubin Observatory) where labeling every object is impossible.
• Robustness: The model's ability to generalize across surveys with vastly different temporal resolutions (days vs. minutes) suggests it learns fundamental physical structures of light curves rather than survey-specific artifacts.
• Practical Impact: By providing pre-trained weights and code, the authors lower the barrier to entry for astronomers to apply deep learning to light curve analysis without needing massive computational resources for training from scratch.
• Future Work: The authors plan to integrate multi-modal data (multi-band light curves) and expand pretraining to entire survey datasets to further refine the embeddings.

In summary, Astromer 2 represents a significant step forward in astronomical time-series analysis, proving that foundational models can bridge the gap between data-rich and data-scarce regimes, enabling accurate classification of variable stars with unprecedented efficiency.