Semantic search for 100M+ galaxy images using AI-generated captions

Technical Summary: Semantic Search for 100M+ Galaxy Images Using AI-Generated Captions

Problem Statement
The astronomical community faces a critical bottleneck in exploring the billions of galaxy images generated by modern telescopes. Traditional discovery methods rely on slow, manual labeling campaigns (e.g., Galaxy Zoo) or supervised deep learning models constrained to fixed question sets. These approaches cannot scale to future datasets (e.g., Euclid, LSST) and lack the flexibility to support open-vocabulary queries. Furthermore, existing self-supervised similarity search methods retrieve visually similar images rather than images containing specific physical phenomena, rendering them insufficient for targeted scientific discovery.

Methodology
The authors propose AION-Search, a pipeline designed to create a semantic search engine from completely unlabeled image data. The methodology proceeds in three stages:

1. VLM Caption Generation: The authors benchmarked 14 Vision-Language Models (VLMs) against human annotations from the Galaxy Zoo-DECaLS catalog. They evaluated models based on their ability to generate free-form descriptions that could be parsed into decision-tree answers. GPT-4.1-mini was selected as the optimal model, offering the best trade-off between accuracy (50.1% match rate on Galaxy Zoo questions) and API cost. Using this model, the team generated detailed descriptions for 300,000 randomly selected galaxy images from the Legacy Survey and Hyper Suprime-Cam (HSC). These descriptions were subsequently summarized into single sentences to reduce noise.
2. Contrastive Alignment: To enable scalable search without captioning every image in the catalog, the authors trained a contrastive learning model. They utilized AION-1, an astronomy foundation model pretrained on over 200 million observations, as the image encoder. The text summaries were embedded using text-embedding-3-large. A four-layer residual MLP projection head was trained to align the image and text embeddings into a shared 1024-dimensional semantic space using a symmetric cross-entropy (InfoNCE) loss. The underlying weights of AION-1 and the text encoder were frozen during this process.
3. VLM Re-ranking: To improve retrieval precision for rare phenomena, the authors introduced a post-search re-ranking stage. After an initial retrieval of the top-1,000 candidates via cosine similarity, a VLM (GPT-4.1) scores each candidate based on the user's query. This step leverages increased compute at inference time to refine the ordering of results.

Key Contributions

• Scalable Semantic Search: The system enables flexible, open-vocabulary semantic search for over 100 million galaxy images (specifically Legacy Survey DR10 galaxies) without requiring human annotation for the entire dataset.
• Zero-Shot Performance: The model achieves state-of-the-art zero-shot performance on finding rare phenomena (spiral galaxies, mergers, and gravitational lenses) despite being trained on randomly selected images with no deliberate curation for these specific classes.
• Re-ranking Framework: The introduction of a VLM-based re-ranking method nearly doubles the recall for the most challenging targets (gravitational lenses) in the top-100 results.
• Public Infrastructure: The authors released the code, trained model weights, pre-computed embeddings, and a public web application (hosted on Hugging Face Spaces) that supports text, image, and hybrid search modes.

Results

• Retrieval Performance: AION-Search significantly outperforms similarity-based baselines (including AION-1 variants, AstroCLIP, and DINOv2). On the nDCG@10 metric, AION-Search achieved scores of 0.941 for spirals, 0.554 for mergers, and 0.173 for gravitational lenses, compared to baselines ranging from 0.015 to 0.643.
• Re-ranking Efficacy: Applying GPT-4.1 re-ranking with 5-sample averaging improved nDCG@10 scores across all categories. For gravitational lenses, re-ranking increased the number of confirmed lenses found in the top-100 results from 7 to 13.
• Zero-Shot Classification: On the Galaxy10 benchmark, AION-Search achieved 53.3% accuracy in zero-shot classification, slightly exceeding the performance of the VLM (GPT-4.1-mini) that generated its training data. When a supervised MLP head was added to the frozen AION-Search embeddings, accuracy reached 84.1%, matching the performance of the base AION-1 model, indicating that the contrastive alignment preserved morphological information.
• Scientific Discovery: Using the system, the authors identified 36 new extragalactic stellar stream candidates through iterative hybrid image and text queries. These candidates were confirmed as novel by cross-matching against 11 existing catalogs of tidal features.

Significance and Claims
The paper claims that this work provides the first example in astrophysics of using VLM-generated captions to generate training sets for semantic retrieval. The authors assert that their approach demonstrates that large, unlabeled scientific image archives can be made semantically searchable without human supervision. They position this as a generalizable framework applicable to other fields with large unlabeled image archives, such as Earth observation, microscopy, and materials science. The work is presented as a step toward the vision of "a country of geniuses in a datacenter," where AI agents can accelerate scientific discovery by analyzing vast datasets that exceed human inspection capacity. The authors remain modest regarding limitations, acknowledging that VLM descriptions can contain hallucinations and that the model inherits biases from the training VLM, but argue that these imperfect annotations still provide a sufficient signal for effective contrastive learning.