🔭 astrophysics

BOOM and Babamul: a real-time, multi-survey, optical alert broker system operating at scale

This paper introduces BOOM, a high-performance, real-time alert broker system built on a Rust-based stack with MongoDB, Valkey, and Kafka that achieves a throughput seven times higher than existing ZTF systems, serving as the foundation for the upcoming public LSST alert broker, Babamul.

Original authors: Theophile Jegou du Laz, Michael W. Coughlin, Peter Bachant, Jacob E. Simones, Thomas Culino, Antoine Le Calloch, Sushant Sharma Chaudhary, Xander J. Hall, Tyler Barna, Daniel Warshofsky, Matthew Graha

Published 2026-05-08

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Theophile Jegou du Laz, Michael W. Coughlin, Peter Bachant, Jacob E. Simones, Thomas Culino, Antoine Le Calloch, Sushant Sharma Chaudhary, Xander J. Hall, Tyler Barna, Daniel Warshofsky, Matthew Graham, Mansi M. Kasliwal, Ashish Mahabal, Joshua S. Bloom, Antonella Palmese, Frank J. Masci, Steven L. Groom, Richard Dekany, Reed L. Riddle, George Helou

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the universe is a giant, bustling city that never sleeps. Every night, powerful telescopes act like millions of security cameras, scanning the sky and snapping photos of everything that moves, flickers, or explodes. These cameras are getting so good that they are taking pictures of billions of stars every single night.

The problem? There are too many pictures. If you tried to look at every single photo, you'd never find the one rare, exciting event you're looking for—like a new supernova or a crashing star.

This is where BOOM and Babamul come in. Think of them as the ultimate "Smart Sorting System" for the universe's security footage.

The Problem: Too Much Noise, Too Little Time

Right now, telescopes like the Zwicky Transient Facility (ZTF) take about 100,000 to 1 million "alerts" (photos of something that changed) every night. Soon, a new telescope called the Vera C. Rubin Observatory (LSST) will take 10 times more than that—about 10 million alerts a night.

If astronomers tried to read every single alert manually, they would be buried under a mountain of data. They need a robot that can read the news, filter out the boring stuff, and only hand them the headlines that matter.

The Solution: BOOM (The Super-Worker)

The authors built a new system called BOOM (Burst & Outburst Observations Monitor). Here is how it works, using simple analogies:

1. The Assembly Line (Instead of a Single Worker)
Old systems were like a single person trying to do everything: read the photo, check if it's real, compare it to old photos, run a math test, and write a report. If that person got tired, the whole line stopped.
BOOM is different. It's like a massive factory with specialized teams.

The Intake Team: Grabs the raw photos.
The Detective Team: Checks if the photo is real or just a glitch (like a bird flying in front of the lens).
The Brain Team: Uses advanced math (Machine Learning) to guess what the object might be.
The Filter Team: Checks if the object matches what specific scientists are looking for.
Because these teams work in parallel, BOOM is 7 times faster than the old systems. It can handle the massive flood of data from the new telescopes without breaking a sweat.

2. The "No-Code" Filter Builder
Usually, to tell a computer what to look for, you need to know complex computer code (like Python). This is like asking a scientist to learn a new language just to ask a question.
The authors built a visual filter builder (shown in Figure 4 of the paper). Imagine it like building with LEGO blocks. A scientist can drag and drop blocks like "Is this a star?" or "Is it brighter than X?" to build a custom filter. They don't need to know how to code; they just need to know what they are looking for.

3. The "Teamwork" Feature
This is BOOM's secret superpower. Usually, telescopes work alone. But BOOM can look at data from multiple telescopes at the same time.

Analogy: Imagine one telescope sees a faint shadow, and another sees a bright flash. Alone, they might miss the story. But BOOM combines them instantly, saying, "Hey, these two events happened at the same time in the same spot! This is a big deal!"
This allows scientists to track how fast things change much better than before.

The Result: Babamul (The Public Library)

Once BOOM does the hard work of sorting and filtering, it sends the "good stuff" to Babamul.
Think of Babamul as a public library or a newsstand for astronomers. It takes the filtered alerts and hands them out to the global community.

It organizes the alerts into different "streams" (like different news channels: one for exploding stars, one for asteroids, etc.).
It provides a website where scientists can search for objects, look at their light curves (graphs of how bright they get), and see images.
It even has a special "app" (a Python client) that lets computers automatically grab the alerts they need.

Why This Matters

The paper demonstrates that BOOM can handle the massive data load of the upcoming Rubin Observatory era. It proved this by:

Speed Tests: Showing it can process alerts much faster than previous systems, even with limited computer power.
Real-World Test: They used it to combine data from the ZTF telescope and a DECam camera. They successfully found a young supernova (SN 2025kwy) by seeing how it looked in both cameras, proving the "teamwork" feature works.

In Summary

The universe is sending us a flood of data. BOOM is the high-speed, smart sorting machine that filters the noise. Babamul is the friendly interface that delivers the interesting stories to astronomers, allowing them to discover the universe's rarest events without getting lost in the data. It's a system built to handle the future of astronomy, starting right now.

Technical Summary: BOOM and Babamul Alert Broker System

Problem Statement
The advent of high-throughput wide-field optical surveys, culminating in the Vera C. Rubin Observatory's Legacy Survey of Space and Time (LSST), presents a data processing challenge that exceeds the capacity of current alert broker systems. While the Zwicky Transient Facility (ZTF) generates $\sim 10^5$ – $10^6$ alerts per night, LSST is projected to produce $\sim 2 \times 10^7$ alerts nightly. Existing brokers, largely built on Python-based stacks (e.g., Kowalski), struggle to scale efficiently to these volumes while maintaining the low-latency requirements necessary for real-time follow-up of fast transients. Furthermore, the scientific community requires the ability to perform joint filtering across multiple overlapping surveys (e.g., ZTF, LSST, and others) to characterize the evolution of fast phenomena, a capability not natively supported by single-survey architectures.

Methodology and System Architecture
The authors introduce BOOM (Burst & Outburst Observations Monitor), a real-time, multi-survey alert broker designed for horizontal scalability and high throughput. The system represents a fundamental architectural shift from its predecessor, Kowalski, by moving away from a monolithic Python-based design to a compiled Rust-based software stack.

Key architectural components include:

Ingestion and Messaging: The system utilizes Apache Kafka for ingesting Avro-formatted alert streams from various surveys. To handle inter-process communication without the I/O bottlenecks of disk-based topics, BOOM employs Valkey (a high-performance, in-memory datastore) as a job queue. This allows multiple worker processes to consume alerts concurrently from memory-resident lists.
Data Storage: MongoDB serves as the primary database, chosen for its flexibility, cross-language support, and native GeoJSON indexing for spatial queries (cone-searches). The data model separates data into three collections:
- Alert: Contains candidate metadata and the latest detection.
- Object: Aggregates time-series data (lightcurves) and cross-matches for unique astronomical objects, avoiding duplication across alerts.
- Cutout: Stores science, reference, and difference image cutouts.
Parallelized Processing Pipeline: BOOM decomposes the alert processing workflow into dedicated, parallelizable worker types:
1. Alert Ingestion Worker: Deserializes Avro packets, performs cross-matching against static archival catalogs and dynamic object collections from other supported surveys, and populates the database.
2. Enrichment Worker: Processes batches of alerts to compute machine learning (ML) scores and pre-computed features. It utilizes ONNX (Open Neural Network eXchange) to run Python-trained models (e.g., ACAI, BTSbot) within the Rust pipeline. The system also supports optional GPU acceleration for batch inference and lightcurve fitting (using Gaussian Processes and Particle Swarm Optimization).
3. Filter Worker: Executes user-defined filters (defined as MongoDB aggregation pipelines) on batches of enriched alerts. It dynamically constructs lookup stages to retrieve necessary data from the Object collection, ensuring efficient resource usage.
User Interface and API: A RESTful API allows for archival searches and the re-running of filters on historical data. To lower the barrier to entry for astronomers, a visual, block-based Filter Builder (integrated with SkyPortal) enables the creation of complex queries without requiring knowledge of the underlying database query language.

Key Contributions

Multi-Survey Joint Filtering: BOOM is designed to ingest and jointly filter multiple alert streams simultaneously. By cross-matching alerts from different surveys (e.g., ZTF and DECam/LSST) in real-time, it enables the construction of joint lightcurves, addressing the limitations of single-survey cadence and depth.
Performance and Scalability: The migration to Rust and the adoption of an in-memory job queue (Valkey) coupled with batch processing significantly improve throughput. The system demonstrates feature parity with existing ZTF brokers while achieving a throughput approximately 7 times higher.
Modular and Extensible Design: The separation of concerns (Ingestion, Enrichment, Filtering) allows for independent scaling of specific processing stages based on workload bottlenecks.
Babamul: The authors introduce Babamul, a public-facing LSST alert broker built atop BOOM. It provides disjoint, categorized Kafka streams (e.g., separating stellar matches from transient candidates) and includes a Python client for easy integration.

Results

Throughput Testing: In benchmark tests using one night of ZTF data (29,142 alerts), BOOM demonstrated superior scalability compared to Kowalski. With 8 worker processes, BOOM processed alerts at a rate sufficient to handle the projected Rubin Observatory alert rate ( $\sim 333$ alerts/s) with a memory footprint of $\sim 1$ GB, compared to Kowalski's 13 GB for the same configuration.
Joint Survey Validation: A joint experiment involving ZTF and the DECam Transients Survey (DTS) successfully validated the system's ability to cross-match alerts between surveys with different depths and cadences. The system identified 207 multi-survey candidates, including SN 2025kwy, demonstrating the ability to constrain the early-time evolution of transients using combined data.
Storage Estimates: The authors estimate that for a 10-year LSST survey, the total storage requirement for BOOM (including alerts, objects, and a 7-day retention of cutouts) would be approximately 75 TB, a figure derived from conservative scaling of existing ZTF data.

Significance
The paper positions BOOM as a critical infrastructure component for the "Rubin era," enabling the community to manage the exponential increase in alert volume while unlocking new science cases through multi-survey synergy. By providing a scalable, high-throughput framework that supports joint filtering and offers user-friendly tools for filter creation, BOOM aims to facilitate the discovery of rare, fast-evolving transients (such as young supernovae and kilonovae) that would otherwise be missed by single-survey analysis. The system is already operational on ZTF and is being deployed to handle public LSST alerts, serving as the foundation for the Babamul public interface.