🔭 astrophysics

Applying Vision Transformers on Spectral Analysis of Astronomical Objects

This paper demonstrates that fine-tuning pre-trained Vision Transformers on large-scale, real astronomical spectral data converted into 2D images achieves superior stellar classification accuracy and competitive redshift estimation compared to traditional machine learning methods and existing deep learning models, marking the first such application without relying on synthetic inputs.

Original authors: Luis Felipe Strano Moraes, Ignacio Becker, Pavlos Protopapas, Guillermo Cabrera-Vives

Published 2026-05-13

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Luis Felipe Strano Moraes, Ignacio Becker, Pavlos Protopapas, Guillermo Cabrera-Vives

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Idea: Teaching a "Picture Expert" to Read "Sound Waves"

Imagine you have a super-smart AI that has spent its whole life looking at millions of photos of cats, dogs, and cars. It is an expert at recognizing patterns in images. Now, imagine you have a pile of sound waves (specifically, light waves from stars and galaxies) that look like squiggly lines on a graph.

Traditionally, to analyze these squiggly lines, scientists had to use math-heavy tools or teach the AI from scratch. This paper asks a simple question: "What if we trick the image-expert AI into thinking these squiggly lines are actually pictures?"

The authors did exactly that. They took one-dimensional data (a single line of light intensity) and turned it into two-dimensional images. Then, they fed these images to a Vision Transformer (ViT)—a type of AI originally designed to recognize images—to see if it could learn to identify stars, galaxies, and quasars, and measure how fast they are moving away from us.

The Problem: The "Template" Bottleneck

Think of traditional astronomy analysis like trying to identify a song by humming a few notes and matching them to a library of pre-recorded tunes.

The Old Way: Scientists take a new star's light spectrum and try to match it against a library of "templates" (pre-defined shapes of light). If the star is weird, noisy, or doesn't fit the template perfectly, the match fails or is inaccurate.
The Issue: This is like trying to fit a square peg in a round hole. It's rigid, and if the data is a bit "fuzzy" (low signal-to-noise), the method struggles.

The Solution: Turning Lines into "Topographic Maps"

The authors realized that Vision Transformers are great at seeing the "big picture" and the "small details" at the same time. To make the AI work, they had to translate the data into a language the AI understood: images.

They tried three different ways to turn the light spectrum into a picture:

The "Simple Plot": Just drawing the line graph as is. (Like showing a raw sketch).
The "Overlap Plot": Cutting the line into three pieces and stacking them as Red, Green, and Blue channels. (Like folding a long map into a colorful tri-fold brochure).
The "2D Map" (The Winner): This was the most creative. They took the long, thin line of data and "squished" it into a square image, filling it like a heatmap. Imagine taking a long strip of wallpaper and rolling it up into a tight, colorful square tile. This allowed the AI to see the entire spectrum as a single, dense image where patterns jumped out visually.

The Training: "SLOMOST"

To teach the AI, they needed a massive library of examples. They combined data from two giant telescopes:

SDSS: A telescope that has mapped millions of objects.
LAMOST: A telescope focused heavily on stars in our own galaxy.

They combined these into a massive dataset they called "SLOMOST" (Sloan + LAMOST). They used over 14 million spectra to train the model. Think of this as showing the AI 14 million different "pictures" of starlight so it could learn the difference between a star, a galaxy, and a quasar on its own.

The Results: The AI Got It Right

The results were surprisingly good. The AI, which was originally trained on photos of everyday objects, adapted perfectly to these "light maps."

Classification (What is it?): The AI had to guess if an object was a Star, a Galaxy, or a Quasar.
- The Result: It got it right 99% of the time. This is better than older methods like Support Vector Machines or Random Forests. It was especially good at telling the difference between galaxies and quasars, which are often confused.
Redshift (How fast is it moving?): Redshift tells us how far away an object is and how fast the universe is expanding.
- The Result: The AI's predictions were incredibly accurate (an $R^2$ score of 0.992). It was as good as, or better than, other top-tier AI models, even though it was trained on a much larger and more diverse set of data.
Stellar Parameters (What is it made of?): They tried to guess the temperature and composition of stars.
- The Result: It worked well, though the "Simple Plot" (just the line graph) actually worked slightly better than the fancy "2D Map" for these specific details. This suggests that for some very subtle tasks, a direct view is better than a transformed one.

Why This Matters

The paper claims this is the first time a Vision Transformer has been applied to real, large-scale astronomical data without using fake (synthetic) data to train it.

The Analogy: It's like taking a chef who only knows how to cook Italian food (ImageNet training) and showing them a few thousand recipes for Japanese cuisine (Spectral data). Instead of teaching them Japanese cooking from scratch, you just show them how to arrange the ingredients on a plate so the chef recognizes the patterns. The chef (the AI) figures out the rest.

Summary of Claims

Method: Convert 1D light spectra into 2D images.
Tool: Use a pre-trained Vision Transformer (ViT).
Data: Real data from SDSS and LAMOST (14M+ objects).
Outcome: The model outperforms traditional machine learning methods in classifying objects and estimating redshift, proving that "image-based" AI can be a powerful tool for reading the light of the universe.

The authors conclude that this approach offers a flexible, scalable way to handle the massive amounts of data coming from future telescopes, turning complex light waves into pictures that AI can easily "see" and understand.

Technical Summary: Applying Vision Transformers on Spectral Analysis of Astronomical Objects

Problem Statement
Astronomical spectroscopy is fundamental for determining the physical and chemical properties of celestial objects and understanding the large-scale structure of the universe. However, the analysis of spectral data faces significant challenges due to the exponential growth in data volume from modern surveys like the Sloan Digital Sky Survey (SDSS) and the Large Sky Area Multi-Object Fiber Spectroscopic Telescope (LAMOST). Traditional methods, such as template-fitting, often simplify data complexity, limiting precision and relying heavily on the selection of reference templates. While manual inspection offers an alternative, it is prone to uncertainty and does not scale to millions of objects. Existing deep learning approaches have shown promise but often rely on simulated data, treat spectra strictly as 1D signals without leveraging multi-channel image processing capabilities, or require specific architectural designs that may not generalize well across diverse object types.

Methodology
The authors propose a novel framework that adapts Vision Transformers (ViTs)—architectures originally designed for image recognition—to analyze one-dimensional astronomical spectra. The core innovation lies in transforming 1D spectral data (wavelength vs. flux) into two-dimensional image representations, thereby enabling the use of pre-trained ViT models.

Data Sources: The study utilizes real spectroscopic data from SDSS (Data Release 18) and LAMOST (Data Release 10, Version 2.0). The datasets were processed to remove low-quality entries and normalized. The authors constructed "Medium" (balanced class representation) and "Big" (full survey volume) datasets, combining them into a joint dataset termed "SLOMOST."
Image Representations: Three distinct methods were explored to encode spectra into 224 × 224 pixel images:
1. Simple Plot: A direct 2D line plot of the spectrum.
2. Overlap Plot: The spectrum is divided into three equal segments, each mapped to a separate RGB color channel to increase detail visibility.
3. 2D Map: A heatmap-like representation where the 1D flux data is reshaped into a square grid (using 3×3 pixel blocks per data point) and colored by intensity, creating a dense spatial encoding of spectral features.
Model Architecture: The authors employed a pre-trained ViT backbone (DINO, specifically facebook/dino-vitb16) originally trained on ImageNet. This model was fine-tuned on the spectral image datasets without pre-training on spectra from scratch.
- Classification: A custom head was added for morphological classification (Stars, Quasars, Galaxies).
- Regression: A linear regression head was attached to the CLS token for estimating redshift ( $z$ ) and stellar parameters (Effective Temperature $T_{eff}$ , Surface Gravity $\log g$ , and Metallicity $[Fe/H]$).
Training: Models were fine-tuned using Mean Squared Error (MSE) for regression and categorical cross-entropy for classification, with hyperparameters optimized via tuning.

Key Contributions

First Large-Scale ViT Application to Real Spectra: To the authors' knowledge, this is the first application of Vision Transformers to large-scale astronomical spectral data using real observations rather than synthetic inputs.
Cross-Modal Adaptation: The work demonstrates the feasibility of repurposing image-based foundation models for 1D scientific data through spatial transformation, leveraging the self-attention mechanism to capture both local and global spectral features.
Comprehensive Benchmarking: The study evaluates multiple encoding strategies and compares performance across diverse tasks (classification, redshift regression, and stellar parameter regression) against established baselines.

Results

Classification: The model achieved high accuracy in distinguishing between stars, galaxies, and quasars. The 2D Map representation yielded the best results, reaching 99.4% accuracy and a 0.991 F1-score on the SLOMOST-Big dataset. This outperformed traditional machine learning baselines, including Support Vector Machines (SVM) and Random Forests (RF), which achieved approximately 96% accuracy in comparable studies. The 2D Map representation specifically reduced confusion between quasars and galaxies.
Redshift Estimation: The model demonstrated strong performance in redshift regression, achieving an $R^2$ of 0.992 on the SLOMOST-Big dataset using the 2D Map representation. This performance is comparable to the spectrum encoder of AstroCLIP (which reported $R^2 = 0.990$ ) but was achieved on a substantially larger and more diverse dataset (14M+ objects vs. ~200k) using a single NVIDIA 4090 GPU over nine days. The model also maintained high accuracy across a wide range of Signal-to-Noise (SNR) ratios.
Stellar Parameter Regression: While the model successfully inferred stellar parameters, the results were more variable. The Simple Plot representation performed best for $T_{eff}$ and $\log g$ ( $R^2 \approx 0.79$ ), while the Overlap Plot was superior for metallicity ($[Fe/H]$) on the large dataset ( $R^2 \approx 0.78$ ). The 2D Map, while excellent for classification and redshift, was less effective for these specific parameter regressions, suggesting that different encoding strategies may be optimal for different physical properties.

Significance and Claims
The paper claims that converting spectral data into 2D image representations allows Vision Transformers to effectively capture complex spectral patterns that traditional 1D models or template-fitting methods might miss. The results demonstrate that pre-trained vision models can be successfully fine-tuned for spectroscopic tasks, offering a scalable, flexible, and data-driven alternative to current pipelines.

The authors emphasize that their approach does not rely on synthetic data, addressing a gap in previous deep learning applications in astronomy. They conclude that while the method shows "strong performance and scalability" in classification and redshift estimation, stellar parameter regression presents additional challenges that may require further architectural refinement or hybrid approaches. The work is positioned as a foundational step toward general-purpose tools for spectral science, capable of handling the increasing volume and heterogeneity of data from future surveys like MSE, 4MOST, and DESI.