🔭 astrophysics

AGNFormer I: Reconstruction of AGN spectra using a probabilistic transformer model

This paper introduces AGNFormer, a probabilistic transformer model that successfully reconstructs masked broad emission lines and diverse spectral features in AGN spectra with high precision by leveraging learnable correlations across the entire optical spectrum from the SDSS DR16 Quasar Catalogue.

Original authors: Benedict L. Rouse, Franz E. Bauer, Tomasz RóĊański, Guillermo Cabrera-Vives, Patricia Arévalo, Cristóbal Rodrigo Donoso Oliva, Pavlos Protopapas, Gordon T. Richards, Paula Sánchez-Sáez, Ezequiel Treis

Published 2026-07-23

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Benedict L. Rouse, Franz E. Bauer, Tomasz RóĊański, Guillermo Cabrera-Vives, Patricia Arévalo, Cristóbal Rodrigo Donoso Oliva, Pavlos Protopapas, Gordon T. Richards, Paula Sánchez-Sáez, Ezequiel Treister, Christian Wolf

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the universe as a giant, noisy radio station. Most of the time, we can only tune into a few clear stations, but sometimes the signal is fuzzy, or parts of the broadcast are missing entirely. In the world of astronomy, these "stations" are Active Galactic Nuclei (AGN)—the super-bright hearts of galaxies powered by massive black holes gobbling up gas. When this gas swirls around the black hole, it glows and shoots out light across the entire spectrum, from invisible ultraviolet rays to visible light. Astronomers use telescopes to catch this light and turn it into a "spectrum," which is like a barcode of colors and brightness. This barcode tells us how heavy the black hole is, how fast it's eating, and what the gas around it is doing. But here's the catch: telescopes aren't perfect. Sometimes the signal is too weak, or the telescope can't see certain colors because of the Earth's atmosphere or the limits of the instrument. It's like trying to listen to a song, but someone has cut out the chorus, or the radio is full of static. For a long time, if a piece of the spectrum was missing or too noisy, astronomers had to guess what was there, often throwing away the messy data.

Enter the world of artificial intelligence, specifically a type of brain-like computer program called a "transformer." You might know these from apps that write stories or translate languages. They are experts at finding patterns in long sequences of data, like words in a sentence. The big question this paper asks is: Can we teach a transformer to look at the parts of a galaxy's light that are visible and use them to perfectly guess the parts that are missing or hidden? If we can do this, we wouldn't just be filling in the blanks; we'd be unlocking a treasure trove of information that was previously lost to noise or gaps in our telescopes. It's like having a super-smart detective who can look at a few scattered clues and reconstruct the entire crime scene with incredible accuracy.

The Paper's Story: AGNFormer

In this paper, the authors introduce a new AI model called AGNFormer. Think of AGNFormer as a cosmic "fill-in-the-blanks" wizard. The researchers trained this wizard on a massive library of over 750,000 galaxy spectra from the Sloan Digital Sky Survey (SDSS). Instead of just memorizing the data, they taught the model a game: "Here is a spectrum, but I'm going to hide a big chunk of it (like the part with the bright hydrogen line). Can you guess what that hidden part looks like based on the rest of the light?"

To make this work, the model doesn't just guess a single number; it acts like a cautious scientist. It predicts the most likely brightness (the mean) and how unsure it is about that guess (the variance). This is crucial because it helps the model know when to trust the data and when to be careful, especially in noisy areas. The model learned to connect the dots between the smooth, continuous glow of the galaxy's disk and the specific, sharp spikes of light caused by gas clouds zooming around the black hole.

What They Found

The results are surprisingly good. When the model tried to guess the missing parts of the spectrum:

For the full set of data (including some very noisy, low-quality signals), it could reconstruct the missing broad lines to within 10–16% of the actual brightness.
When they tested it on high-quality data (where the signal-to-noise ratio, or S/N, was greater than 10), the accuracy jumped to 4–8%.
Even when the model had to guess a whole half of the spectrum (like predicting the red side of the rainbow based only on the blue side), it did a great job, with errors growing only slightly the further away the guess was from the known data.

The model was so good that it could even handle different "personalities" of galaxies. Some galaxies have broad lines that look like smooth hills (Gaussian profiles), while others look like sharp spikes (Lorentzian profiles). AGNFormer learned to recognize and recreate both types, even though some of these shapes were rare in the training data. It also successfully predicted complex mixtures of light, like the "Fe II" complexes (a messy bunch of iron lines) and narrow lines, showing that it understood the deep connections between all the different parts of the light.

Beating the Competition

The authors didn't just stop at their own results; they put AGNFormer to the test against other methods. In a blind challenge to predict the Lyman-alpha line (a tricky ultraviolet feature) for 30 high-redshift quasars, AGNFormer outperformed all 11 other algorithms, including specialized tools designed just for this job. It managed to do this even when trained only on lower-resolution data but tested on higher-resolution data, proving it learned the physics of the light, not just the specific numbers of the telescope.

The "Black Hole" of Absorption

There was one tricky situation: some galaxies have "Broad Absorption Lines" (BALs), where gas is moving so fast it blocks out the light, creating dark troughs in the spectrum. The model was trained to ignore these, but when tested on them, it sometimes over-predicted the brightness in the dark areas. This suggests that while the model is amazing at reconstructing the "standard" look of a galaxy, it still struggles a bit with the chaotic, random nature of these high-speed winds. However, compared to older methods, it still provided a very reasonable guess, often capturing the shape of the line better than previous techniques.

Why This Matters

The main takeaway is that AGNFormer proves we can use the "leftover" information in a spectrum—the faint, messy, or seemingly useless parts—to reconstruct the whole picture. It shows that the light from the black hole's disk and the fainter lines are deeply connected, holding secrets that we can now unlock with AI. This isn't just about filling in gaps; it's about realizing that every photon in a spectrum has a story to tell, and with the right tools, we can finally hear the whole song. The authors suggest this model could become a standard tool for astronomers, helping them extract more information from existing data and perhaps even guiding future telescopes on what to look for next.

Technical Summary: AGNFormer I: Reconstruction of AGN spectra using a probabilistic transformer model

Problem Statement
Active Galactic Nuclei (AGN) spectra contain complex, correlated information across the ultraviolet (UV) and optical bands, including the accretion disk continuum, broad emission lines (BLR), narrow emission lines (NLR), and blended complexes (e.g., Fe ii). Traditional analysis often focuses on specific features (e.g., H $\beta$ for black hole mass) while discarding or underutilizing the continuum and weaker lines. Furthermore, observational data often suffers from masked regions due to instrumental gaps, atmospheric absorption, or the need to estimate intrinsic line profiles obscured by Broad Absorption Lines (BALs). Existing reconstruction methods, such as Principal Component Analysis (PCA) or Independent Component Analysis (ICA), often rely on linear assumptions or require specific training on non-BAL samples, limiting their ability to capture non-linear correlations and diverse spectral morphologies across the AGN Main Sequence (QMS).

Methodology
The authors introduce AGNFormer, a self-supervised, probabilistic transformer model designed to reconstruct masked regions of AGN spectra.

Architecture: The model utilizes an encoder-only transformer architecture (based on Vaswani et al. 2017) with specific modifications for astronomical data:
- Wavelength-based Positional Encoding: Instead of discrete token indices, the model uses a continuous positional encoder based on physical wavelength ( $\log_{10}\lambda$ ). This allows the model to handle varying redshifts, irregular wavelength grids, and different spectral resolutions without resampling.
- Probabilistic Output: The model predicts both the mean flux ( $\mu$ ) and the log-variance ( $\log \sigma^2$ ) for masked pixels. It treats the predictive distribution as an independent Gaussian.
- Uncertainty Injection: Input fluxes are perturbed with Gaussian noise scaled by observational uncertainties during training. The model explicitly takes flux uncertainties as input, allowing it to distinguish between genuine spectral features and noise.
- Registers: Randomly initialized "register" tokens are appended to the input sequence to absorb attention artifacts and improve the concentration of attention maps on relevant spectral features.
- Normalization: A modified Euclidean normalization is applied using only unmasked pixels to prevent information leakage from masked regions into the normalization factor.
Training Strategy:
- Data: The model is trained on the SDSS DR16 Quasar Catalogue (750,414 broad-line quasars, $0.1 < z < 7$ ), excluding sources with strong BALs ( $P_{BAL} > 0.0$ ) during training but retaining them for testing.
- Masking: A combination of random pixel masking, random chunk masking, and fixed masking (centered on major broad lines with a width of $\pm 10,000$ km s $^{-1}$ ) is used.
- Loss Function: The model minimizes the heteroscedastic Gaussian negative log-likelihood (NLL). This loss function weights the error by the inverse variance, penalizing incorrect predictions in low-noise regions more heavily than in high-noise regions.

Key Results

Broad Line Reconstruction: AGNFormer successfully reconstructs masked broad lines (O I, Si IV, C IV, C III], Mg II, H $\gamma$ $γ$ , H $\beta$ $β$ , H $\alpha$ $α$ ) with high fidelity.
- For the full test set (including low S/N), the median prediction error is 10–16% of the flux.
- For high-quality spectra (S/N > 10), the error drops to 4–8%.
- An error floor of $\approx$ 2–6% is observed at S/N $\approx$ 40, suggesting the model's intrinsic error dominates over data noise at high S/N.
Spectral Diversity: The model accurately reproduces diverse line profiles across the QSO Main Sequence, including both Lorentzian profiles (low FWHM) and Gaussian profiles (high FWHM), as well as complex Fe ii blends and narrow lines.
Half-Spectrum Prediction: The model can predict an unseen half of a spectrum (blue or red) based on the observed half, effectively extending the wavelength coverage of a single observation.
Comparison to Existing Methods:
- Ly $\alpha$ Reconstruction: In a blind challenge against 11 other algorithms (including PCA and deep learning methods) using 30 X-Shooter and 30 SDSS spectra, AGNFormer achieved the lowest Mean Absolute Error (MAE) and median $\sigma$ , outperforming all competitors. Notably, it achieved this without fine-tuning, demonstrating resolution invariance between SDSS ( $R \sim 2000$ ) and X-Shooter ( $R \sim 4500-7500$ ) data.
- BAL Reconstruction: When tested on BAL QSOs (which were excluded from training), the model generally outperformed the MFICA method (Rankine et al. 2020) in terms of dispersion symmetry and median accuracy, though it occasionally overpredicted flux in the presence of strong absorption blueward of C IV.
Calibration: The "pull" distributions (residuals normalized by predicted and observed errors) closely follow a standard normal distribution $N(0, 1)$ , indicating that the model's uncertainty estimates are well-calibrated.

Significance and Claims
The paper claims that AGNFormer demonstrates the efficacy of transformer-based architectures in leveraging the entire spectral context of AGN, rather than isolated features. Key contributions include:

Information Aggregation: The model proves that the AGN continuum and weaker lines contain sufficient information to reconstruct major broad lines with high precision, validating the existence of strong, learnable correlations across the spectrum.
Uncertainty Awareness: By explicitly modeling and predicting flux uncertainties, the model avoids overconfidence in noisy regions and effectively weights high-S/N data, leading to superior performance compared to deterministic models.
Generalizability: The use of physical wavelength encoding allows the model to generalize across different redshifts and instrumental resolutions without retraining.
Foundation for Downstream Tasks: The authors posit that the learned representations and attention mechanisms provide a robust foundation for future supervised tasks, such as estimating black hole masses, Eddington ratios, and other physical parameters, as well as outlier detection.

The work establishes a new benchmark for spectral reconstruction in astronomy, moving beyond linear decomposition methods to non-linear, probabilistic deep learning approaches that respect the physical and statistical properties of observational data.

More like this