The Gravitational-wave Optical Transient Observer (GOTO) data pipeline and workflow for transient discovery

This paper describes the development, implementation, and performance assessment of a low-latency data pipeline and workflow for the Gravitational-wave Optical Transient Observer (GOTO) that enables the rapid discovery, reporting, and characterization of astrophysical transients within approximately seven minutes of observation.

The Gist

Imagine the night sky as a giant, bustling city that never sleeps. Every night, new things happen: stars explode, black holes eat asteroids, and distant galaxies flare up. These events are like "transients"—they appear suddenly, shine brightly for a short time, and then vanish. To catch them, you need a security system that watches the whole city, 24/7, and alerts the police the second something suspicious happens.

This paper describes GOTO (Gravitational-wave Optical Transient Observer), which is exactly that security system, but for the universe. However, a security system is useless if the camera footage takes days to process. This paper explains the high-speed data pipeline that GOTO built to catch these cosmic events in real-time.

Here is how the system works, broken down into simple analogies:

1. The Eyes: A Swarm of 32 Cameras

GOTO isn't just one big telescope; it's a swarm of 32 small cameras (40cm telescopes) split between two locations: one in the Canary Islands and one in Australia.

• The Analogy: Think of it like a security team with 32 different cameras covering different neighborhoods. Because they are spread out, they can watch the sky almost all night long, no matter where the sun is.
• The Job: They do two things:
  1. The Patrol: They scan the whole sky regularly to find new things on their own.
  2. The Emergency Response: When a "911 call" comes in from other observatories (like a gravitational wave detector or a gamma-ray burst alert), GOTO immediately swings its cameras to that specific spot to see what caused the alarm.

2. The Data Highway: From Camera to Computer

When a camera takes a picture, the data is huge. It needs to get from the telescope (in Spain or Australia) to the processing center (in the UK) instantly.

• The Analogy: Imagine the cameras are like delivery trucks dropping off packages. The rawtransfer system is the high-speed conveyor belt that grabs the package the millisecond it hits the dock and ships it to the warehouse in the UK.
• The Speed: It takes less than a minute for the data to travel across the ocean and land on the UK servers.

3. The Assembly Line: The "Kadmilos" Pipeline

Once the data arrives, it needs to be cleaned and analyzed. The authors call this software kadmilos.

• The Analogy: Think of kadmilos as a super-fast car wash and inspection line.
  • Step 1: The Wash (Calibration): The raw photos are dirty. They have dust on the lens (dead pixels), uneven lighting (vignetting), and weird electrical noise. The system uses "Super Calibration" frames (like a master template of a clean lens) to scrub the image clean.
  • Step 2: The "Column Trap" Fix: Sometimes, electrons get stuck in the camera's sensor, creating a vertical streak of garbage in the image. The system has a special tool to identify these "stuck" columns and digitally erase them.
  • Step 3: The "Subtract" Trick (Difference Imaging): This is the magic trick. To find a new star, you don't just look at the new photo; you take the new photo and subtract an old photo of the exact same spot.
    • Result: Everything that stays the same (static stars, galaxies) disappears. Only the new things (the transient) remain. It's like taking a photo of a room, waiting an hour, taking another photo, and then using Photoshop to remove everything that didn't move. The only thing left is the person who walked in.

4. The Traffic Cop: Apache Airflow

Managing 32 cameras sending data every minute is chaotic. You need a traffic cop to make sure the right tasks happen in the right order.

• The Analogy: Apache Airflow is the conductor of a massive orchestra. It tells the computers: "First, clean the image. Then, find the stars. Then, subtract the old photo. Then, check if it's real." If one instrument (computer) breaks, the conductor reroutes the music so the show goes on without missing a beat.

5. The Detective: The "Marshall"

Once the system finds a "new" dot of light, it needs to decide: Is this a real cosmic event, or just a satellite, a bird, or a glitch?

• The Analogy: The GOTO Marshall is the detective's office.
  • The Inbox: New candidates land here.
  • The Filter: The system automatically checks: "Is this a known satellite?" (Trash). "Is this a known variable star?" (File away). "Is this near a galaxy?" (Keep).
  • The Human Touch: If the computer is unsure, it puts the image in the "Inbox" for a human astronomer to look at.
  • The "Citizen Science" Link: They also use a project called Kilonova Seekers, where regular people (like you and me) look at tiny image snippets and vote "Real" or "Fake." The system uses these votes to help decide what to report.

6. The Alarm: Reporting and Follow-up

If the system (and the humans) agree it's a real transient, it needs to be reported to the world immediately.

• The Analogy: The Marshall sends a GCN Circular (a global alert) to every other telescope on Earth.
• The "Auto-Trigger": If the object looks really exciting (like a potential kilonova), the system can automatically send a request to other telescopes (like the Liverpool Telescope) to zoom in and take a spectrum (a chemical fingerprint) of the object, all without a human pressing a button.

The Result: Speed is Everything

The paper highlights that from the moment the telescope shutter closes to the moment a human sees a candidate on their screen, it takes about 7 minutes.

• Why this matters: In astronomy, "infant" transients (like the very first flash of a supernova) fade quickly. If you wait a day, you miss the most interesting part of the story. GOTO's pipeline allows scientists to catch these events while they are still "babies," giving us a front-row seat to the death of stars and the birth of black holes.

Summary

This paper is a blueprint for a cosmic emergency response team. It combines a swarm of cameras, a high-speed data highway, a smart cleaning crew, a detective system, and a global alert network to ensure that when the universe has a "breaking news" story, we are the first to know.

Technical Summary

1. Problem Statement

The discovery and characterization of astrophysical transients (e.g., supernovae, kilonovae, tidal disruption events) require wide-field, high-cadence sky surveys capable of rapid response to external multi-messenger triggers (such as gravitational waves or gamma-ray bursts).

• The Challenge: Identifying optical counterparts to these triggers requires low-latency data processing to distinguish real astrophysical signals from noise, cosmic rays, and instrumental artifacts.
• The Gap: Existing pipelines often struggle to balance the speed required for "infant" transient characterization (minutes after observation) with the robustness needed for high-quality photometry and astrometry across a large, distributed telescope array.
• The Context: The GOTO facility, a 32-telescope array split between La Palma (Spain) and Siding Spring (Australia), needed a custom end-to-end workflow to handle data from shutter close to community reporting and automated follow-up.

2. Methodology and System Architecture

The paper details the kadmilos pipeline and the GOTO marshall workflow, designed to process data from 32 unit telescopes (UTs) into transient candidates within minutes.

A. Infrastructure and Data Flow

• Hardware: 32 x 40cm telescopes (4x2 grid per mount) with ~50M pixel CCDs.
• Data Transfer: Raw FITS files are transferred via rawtransfer (SSH-based) from observatories to a central data center at the University of Warwick.
• Orchestration: Apache Airflow manages the workflow using Directed Acyclic Graphs (DAGs). It handles task scheduling, dependency management, and monitoring via Celery workers.
• Storage: PostgreSQL serves as the central database for metadata, photometry, and image statistics, utilizing Q3C spatial indexing for rapid coordinate queries.

B. The kadmilos Pipeline (Data Reduction)

The pipeline processes data in a low-latency ETL (Extract-Transform-Load) model:

1. Raw Data Transfer & Ingestion: FITS files are transferred and validated.
2. Calibration (Super Calibration Frames):
   • Instead of processing individual bias/dark/flat frames for every image, the system generates monthly super calibration frames (stacks of bias, dark, and flat images).
   • Column Trap Correction: A novel step identifies and corrects "column traps" (charge trapping in CCD columns) using a step-function model fitted to stacked images. This is crucial for GOTO's specific sensor characteristics.
3. Single Image Processing:
   • CCD Reduction: Bias subtraction, dark correction, flat-fielding, and column trap removal.
   • Cosmic Ray Removal: Iterative application of the astro-SCRAPPY algorithm (Laplacian kernel).
   • Background Subtraction: Uses sep (Source Extractor Python) with sigma-clipping to model and subtract spatially varying sky backgrounds.
   • Source Detection & Photometry: Detects sources >2.5$\sigma$ above background. Uses Radial Basis Functions (RBFs) to model aperture corrections and spatially varying zeropoints.
   • Astrometry: Solves World Coordinate Systems (WCS) using astrometry.net and cross-matches with the ATLAS-REFCAT2 catalogue, applying Simple Image Polynomials (SIP) for distortion correction.
4. Stacking (Set Images): Single images are combined into "set" images (weighted by transparency and FWHM) to increase depth.
5. Difference Image Analysis (DIA):
   • Subtracts a high-quality template image (12x90s stack) from the set image using HOTPANTS.
   • Candidate Filtering: Candidates undergo forced photometry on the original set image and constituent singles.
   • Real/Bogus Classification: A Convolutional Neural Network (CNN) trained on GOTO data assigns a "realbogus" score. This score is modulated by the fraction of single images showing a flux excess ($f_{single}$) to filter out residual cosmic rays.

C. The GOTO Marshall (Vetting and Reporting)

The marshall is a web-based interface (Django/Celery) for human-in-the-loop vetting and automation.

• Ingestion: Filters candidates based on "realbogus" scores (thresholds vary by survey mode: >0.7 for sky-survey, >0.4 for follow-up).
• Source Association: Links new detections to existing sources using HEALPix indices to prevent race conditions.
• Automated Vetting:
  • Cross-matches with catalogues (Gaia, TNS, LSXPS, GLADE+, NED).
  • Identifies Solar System bodies using pympc (Minor Planet Center ephemerides) to flag moving objects as "Junk."
  • Classifies sources into lists: Pending, Inbox, Stream (transients), Store (variables/AGN), Junk, and Banish.
• External Trigger Cross-Matching: Independently cross-matches sources against gravitational wave (GW) and GRB skymaps (using MOC formats) to identify counterparts, even if the telescope wasn't explicitly scheduled for that trigger.
• Automated Follow-up (GOAT): The GOTO Auto-Trigger system automatically schedules observations on external facilities (e.g., Liverpool Telescope) if a source meets criteria (e.g., rising light curve, extragalactic location, high SNR).

3. Key Contributions

1. End-to-End Low-Latency Pipeline: A fully automated system capable of processing data from shutter close to candidate generation in ~7 minutes (median).
2. Column Trap Correction: Development of a specific calibration method for GOTO's CCDs to mitigate charge trapping artifacts that standard pipelines miss.
3. Hybrid Vetting Strategy: A robust workflow combining machine learning (CNN for real/bogus), heuristic filtering (Solar System ephemerides), and human-in-the-loop vetting, integrated with citizen science (Kilonova Seekers).
4. Independent Trigger Cross-Matching: A system that decouples transient discovery from the telescope scheduler, ensuring serendipitous discoveries in GW/GRB fields are not missed due to scheduling delays.
5. Automated Follow-up Triggering: Implementation of the GOAT system to autonomously trigger spectroscopic and imaging follow-up on external telescopes within ~11 minutes of shutter close.

4. Results and Performance

• Processing Speed:
  • Data transfer from La Palma: $P_{50} = 9$ seconds.
  • Total pipeline processing (shutter close to candidate): Median ~7 minutes.
  • Automated follow-up trigger (shutter close to LT request): Median ~11 minutes.
• Discovery Rate:
  • As of Jan 2026, GOTO has reported 4,623 first discoveries and 11,800 total candidates to the Transient Name Server (TNS).
  • Reporting rate increased from ~2/day to 15–16/day after the deployment of auto-vetting and improved template coverage.
• Data Quality:
  • Astrometry: Residuals of 0.32–0.38 arcseconds (approx. 1/3 of a pixel).
  • Photometry: 5$\sigma$ depth of $m_L \sim 20$ mag for sky-survey and $\sim 20.5$ mag for follow-up (4x90s stacks).
  • Real/Bogus: The CNN model effectively filters artifacts, though persistent residuals around bright stars remain a challenge.
• Follow-up Success: Confirmed optical afterglows for GRBs (including those with degree-scale localizations) and kilonova candidates, demonstrating the system's efficacy in rapid characterization.

5. Significance

This paper presents a mature, production-grade data pipeline for a next-generation transient survey facility. Its significance lies in:

• Scalability: Successfully handling the continuous, high-volume data stream of a 32-telescope array using open-source tools (Airflow, Celery, PostgreSQL).
• Speed vs. Accuracy: Achieving a balance where data is processed fast enough to characterize "infant" transients (critical for kilonovae and GRB afterglows) without sacrificing the photometric and astrometric precision required for scientific analysis.
• Community Integration: By automating the reporting to TNS and enabling rapid follow-up, GOTO acts as a critical node in the global multi-messenger network, bridging the gap between high-energy triggers and optical characterization.
• Future-Proofing: The architecture is designed to evolve, with planned upgrades to Airflow 3.0 for better versioning and integration of advanced machine learning models for moving body identification and classification.

The workflow described serves as a blueprint for future time-domain facilities (like the Vera C. Rubin Observatory) that must manage massive data volumes while maintaining rapid response times to transient events.