The Sound of Noise: Leveraging the Inductive Bias of Pre-trained Audio Transformers for Glitch Identification in LIGO

Technical Summary: The Sound of Noise: Leveraging the Inductive Bias of Pre-trained Audio Transformers for Glitch Identification in LIGO

Problem Statement
Transient noise artifacts, known as "glitches," fundamentally limit the sensitivity of gravitational-wave (GW) interferometers like LIGO. These artifacts, arising from environmental coupling and instrumental instabilities, can mimic true astrophysical signals, particularly short-duration intermediate-mass black hole (IMBH) mergers. Current classification methods, such as Gravity Spy, rely on supervised Convolutional Neural Networks (CNNs) trained from scratch on massive, expertly labeled datasets. These approaches face a "label bottleneck," requiring extensive manual annotation to achieve high accuracy and often struggling to generalize to new glitch morphologies or evolving detector configurations between observing runs (e.g., O3 to O4). Furthermore, supervised models are inherently reactive, necessitating expensive retraining campaigns when glitch characteristics change.

Methodology
The authors propose a cross-domain framework that treats GW strain data as an audio processing problem. Instead of training a model from scratch, they leverage the Audio Spectrogram Transformer (AST), a vision-transformer model pre-trained on millions of hours of natural audio (AudioSet). The core hypothesis is that the spectro-temporal features learned from natural sound transfer effectively to GW detector noise, providing a strong inductive bias that facilitates rapid learning with minimal labeled data.

The technical workflow involves:

1. Preprocessing: Raw GW strain data is converted into log-mel spectrograms (1024 time frames × 128 frequency bins).
2. Feature Extraction: The spectrograms are processed by the AST encoder. The 2D input is split into 16×16 patches, linearly projected, and passed through 12 Transformer layers containing Multi-Head Self-Attention mechanisms.
3. Parameter-Efficient Fine-Tuning (PEFT): To adapt the pre-trained model to LIGO data without catastrophic forgetting or prohibitive computational costs, the authors employ Low-Rank Adaptation (LoRA). This technique freezes the majority of the pre-trained weights and updates only a small subset of low-rank matrices, steering the model's audio inductive bias toward glitch morphology.
4. Dimensionality Reduction and Clustering: The resulting 768-dimensional semantic embeddings are reduced via Principal Component Analysis (PCA) and then projected into a 3D latent space using t-Distributed Stochastic Neighbor Embedding (t-SNE) for visualization and unsupervised clustering.

Key Contributions and Results
The study validates this approach using data from LIGO's third (O3) and fourth (O4) observing runs:

• Superior Feature Extraction via Inductive Bias: Comparisons between a "vanilla" (off-the-shelf) AST encoder and a LoRA-fine-tuned version demonstrate that pre-training provides a useful baseline. The vanilla encoder already separates high-SNR, visually distinct glitch families (e.g., "Blip," "Whistle," "Koi Fish") reasonably well. However, fine-tuning significantly sharpens the feature space, particularly for low-frequency and morphologically heterogeneous glitches (e.g., "Low-Frequency Burst," "Fast Scattering") that the vanilla model struggled to separate.
• Quantitative Improvements: The fine-tuned model yields higher Silhouette scores across all tested classes, indicating tighter, more compact clusters with clearer boundaries. The most significant gains were observed in low-frequency glitch families, which previously exhibited near-zero or negative scores due to domain mismatch.
• Signal-Agnostic Anomaly Detection: In a zero-shot experiment, the model—fine-tuned only on glitch data with no exposure to GW waveforms—successfully separated injected astrophysical signals (both standard BBHs and short-duration IMBHs) from noise clusters. The GW signals formed a distinct, compact cluster separate from the glitch manifold, suggesting the framework can identify anomalous transients without supervised training on specific waveform families.
• Unsupervised Clustering Validation: Agglomerative clustering on the AST embeddings of 21 glitch types across observing runs recovered 18 distinct clusters. The results showed strong agreement with Gravity Spy classifications, while also revealing natural mergers of morphologically similar classes and splits of single labels into sub-populations, reflecting intra-class diversity or evolution across runs.
• Cross-Run Generalization: When O4 Omicron triggers were projected into the O3-trained embedding space, most triggers mapped to their corresponding O3 clusters. However, certain classes (e.g., "Tomte") showed dispersion, indicating potential morphological evolution or labeling ambiguity in O4, demonstrating the framework's utility for monitoring distribution shifts.

Significance
The paper claims that leveraging the inductive bias of pre-trained audio transformers offers a robust, data-efficient pathway for detector characterization. By shifting from training domain-specific models from scratch to adapting foundation models, the method mitigates the "label bottleneck" inherent in supervised CNNs. The framework enables:

1. Rapid Adaptation: The use of LoRA allows for efficient re-tuning as detector conditions evolve, requiring only modest, high-purity labeled sets.
2. Discovery of Novel Transients: The unsupervised nature of the embedding space allows for the identification of anomalous events that lie outside the manifold of known glitch morphologies, which is critical for low-latency operations in next-generation detectors.
3. Generalization: The ability to distinguish between complex noise artifacts and true astrophysical signals (including IMBHs) without explicit training on those signals suggests a new paradigm for signal-agnostic anomaly detection.

The authors conclude that this approach provides a scalable solution for the increasing complexity of GW data analysis, particularly as detectors move toward the O5 era and beyond.