Leveraging Multimodality for Real-Time Classification of Transients and Variables found by the Zwicky Transient Facility

Technical Summary: Leveraging Multimodality for Real-Time Classification of Transients and Variables found by the Zwicky Transient Facility

Problem Statement
Modern time-domain surveys, such as the Zwicky Transient Facility (ZTF) and the upcoming Vera C. Rubin Observatory's Legacy Survey of Space and Time (LSST), generate hundreds of thousands to millions of alerts nightly. A critical bottleneck in this data deluge is the need for robust, real-time classification to prioritize sources for scarce spectroscopic follow-up resources. Early-time observations are often characterized by sparse light curves and degeneracies between classes, making fine-grained classification difficult. While machine learning has been adopted to address this, relying solely on light curves often yields suboptimal performance at early epochs. Furthermore, existing systems often lack the flexibility to provide classifications at varying levels of granularity as more data becomes available.

Methodology
The authors introduce ORACLE-2, a family of multimodal, hierarchical classifiers designed to improve real-time classification by leveraging multiple data modalities: light curves, tabular metadata, and images. The work builds upon the Online Ranked Astrophysical CLass Estimator (ORACLE) framework, which organizes classes into a taxonomy progressing from coarse (Transient vs. Persistent) to granular categories (e.g., specific supernova subtypes).

The study develops three specific model variants:

1. ORACLE-2 Lite: Utilizes light curves only.
2. ORACLE-2: Combines light curves with tabular metadata.
3. ORACLE-2 Omni: Integrates light curves, metadata, and images.

Architectural Details:

• Time Series Backbone: Uses Gated Recurrent Units (GRUs) to process light curves. Unlike previous versions, ORACLE-2 introduces an attention pooling mechanism over the entire sequence to aggregate hidden states into a fixed-dimensional context vector, mitigating the difficulty RNNs face with long-term dependencies in high-cadence surveys.
• Metadata Backbone: Employs a Multi-Layer Perceptron (MLP) to process tabular features (e.g., sky location, host galaxy properties, AllWISE magnitudes).
• Image Backbone: Utilizes a ConvNeXt-Pico convolutional neural network (CNN), pre-trained on the Galaxy Zoo dataset, to process reference images (either single-channel ZTF or triple-channel Pan-STARRS-1). The model implicitly learns host galaxy associations without explicit feature engineering.
• Training Strategy: The models are trained using a Weighted Hierarchical Cross Entropy (WHXE) loss function to handle the taxonomy structure and class imbalance. A three-step training routine is employed for the multimodal Omni model: independent pre-training of branches, a "warm-up" stage with a frozen backbone and a trainable classification head, and finally, full fine-tuning.

Datasets

• ZTF (Real Data): The Bright Transient Survey (BTS) dataset containing ~14,800 spectroscopically labeled sources, including various supernova subtypes (Ia, Ib/c, II, SLSN-I), Active Galactic Nuclei (AGN), Cataclysmic Variables (CVs), and variable stars.
• ELAsTiCC (Simulated Data): A simulated dataset for LSST containing ~50 million observations of ~5 million sources, used to test scalability and performance on future survey volumes.

Key Results

• Multimodal Superiority: Incorporating additional modalities consistently improves classification performance. On the ZTF BTS dataset, the ORACLE-2 Omni model achieves a macro F1 score of 0.73 at 1024 days post-detection. This represents an improvement of up to 11% over models using light curves and metadata alone (ORACLE-2) and up to 40% over light-curve-only models (ORACLE-2 Lite).
• Early-Time Performance: The gains from multimodality are most pronounced at early times (within the first week), where the macro F1 score improvement reaches 9–13%. This is critical for triage decisions when light curve morphology is not yet fully developed.
• Image Modalities: While triple-channel Pan-STARRS-1 images yield slightly better standalone image classification than single-channel ZTF images, the single-channel ZTF variant of the Omni model performs comparably to the PS1 variant in the full multimodal setting. This suggests that light curve morphology and metadata compensate for the lack of multi-band imaging, making the single-channel ZTF approach more practical for real-time deployment.
• LSST Readiness: On the simulated ELAsTiCC dataset, the ORACLE-2 model achieves a macro F1 score of 0.88 at 1024 days, matching or surpassing other state-of-the-art models (e.g., ATCAT, RoMAE) and demonstrating robustness for future high-volume surveys.
• Throughput: The addition of the image branch reduces GPU throughput by a factor of three but only reduces CPU throughput by ~35%. Given that the authors' production pipeline runs on CPUs, the computational overhead is deemed acceptable given the performance gains.

Real-World Deployment
The single-channel ZTF variant of ORACLE-2 Omni was deployed on the BOOM broker for the ZTF alert stream in May 2026. In a real-time monitoring period (May 15 – June 11, 2026), the model processed 344 new sources with spectroscopic labels, achieving a depth-1 macro F1 score of 0.88 and an accuracy of 0.99. It successfully identified rare and interesting transient sources for follow-up.

Significance and Claims
The paper claims that combining multiple modalities (light curves, metadata, and images) significantly enhances early-time classification, enabling more effective triage of high-volume alert streams. Key contributions include:

1. Practical Multimodal Framework: Demonstrating that multimodal approaches are not only theoretically superior but practically deployable in real-time broker environments.
2. Hierarchical Flexibility: Providing a flexible framework that allows for confident, coarse classifications at early times and granular classifications as data accumulates.
3. Implicit Learning: Showing that deep learning models can implicitly learn salient features of the source environment (host galaxies) from images without explicit feature engineering.
4. Open Science: Releasing the ORACLE-2 family of models, image backbones, and hierarchical classification framework as open-weight tools to facilitate integration into other pipelines.

The authors conclude that while challenges remain regarding underrepresented classes and distribution shifts between simulated and real data, multimodal approaches are essential for managing the data volume of current and future time-domain surveys.