🤖 AI

CLiMB: A Domain-Informed Novelty Detection Clustering Framework for Galactic Archaeology and Scientific Discovery

The paper introduces CLiMB, a domain-informed two-phase clustering framework that effectively balances the classification of known galactic structures with the detection of novel anomalies in Gaia data, significantly outperforming existing baselines in both accuracy and scientific discovery capabilities.

Original authors: Lorenzo Monti, Tatiana Muraveva, Brian Sheridan, Davide Massari, Alessia Garofalo, Gisella Clementini, Umberto Michelucci

Published 2026-02-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lorenzo Monti, Tatiana Muraveva, Brian Sheridan, Davide Massari, Alessia Garofalo, Gisella Clementini, Umberto Michelucci

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are an archaeologist digging through a massive, messy pile of ancient artifacts. Most of the pile is just random dirt and rocks (background noise), but buried within are two types of treasures:

Famous Artifacts: You already know what these look like (e.g., "That's definitely a Roman coin"). You have a few examples of them to help you spot the rest.
Mysterious New Treasures: You don't know what these are yet. They might be a completely new type of tool or a lost civilization's jewelry. You need to find them without accidentally mixing them up with the Roman coins.

The Problem with Old Methods:
Traditional computer programs for sorting these artifacts usually do one of two things, and both fail in this specific scenario:

The "Blind Sorter": It ignores your knowledge of the Roman coins and tries to group everything based on shape alone. It often misses the subtle differences between the famous coins and the new treasures.
The "Strict Librarian": It listens to you about the Roman coins but gets too strict. If you say "Group these coins together," it might force a brand-new, weird-looking artifact into that group just because it's slightly similar, effectively hiding the new discovery.

The Solution: CLiMB
The paper introduces a new framework called CLiMB (CLustering in Multiphase Boundaries). Think of CLiMB as a two-step expert team designed to solve this exact problem.

Phase 1: The "Anchored Anchor" (K-Bound)

Imagine you have a map of the known Roman coins. You drop heavy anchors on the map where you know these coins are.

How it works: The computer looks at the data and says, "Okay, I see these anchors. I will build a tight, protective fence around them."
The Magic: It uses a flexible fence (called a Mahalanobis distance). Imagine the coins aren't in a perfect circle, but in a stretched-out oval shape (like a football). A normal fence would be a circle and cut off the ends. CLiMB's fence stretches to fit the exact shape of the known coins.
The Result: It successfully gathers all the known Roman coins into neat groups without letting any weird, unknown junk sneak inside.

Phase 2: The "Wild Explorer" (Density-Based)

Now, look at everything that didn't get caught in the first fence. This is the "residual" pile—the stuff the computer couldn't confidently label as a Roman coin.

How it works: The computer puts on its explorer hat and looks at this leftover pile. It ignores the old rules about Roman coins. Instead, it just looks for "clumps" or "densities." "Hey, look! There's a big clump of weird shapes over here that looks different from the dirt!"
The Result: Because the computer didn't try to force these new shapes into the "Roman coin" category, it can spot entirely new patterns.

The Real-World Test: Galactic Archaeology

The authors tested this on real data from the Gaia Space Telescope, which maps stars in our Milky Way galaxy.

The Goal: Find "stellar streams"—groups of stars that were swallowed by our galaxy from smaller galaxies billions of years ago.
The Knowns: They knew where 8 specific ancient star groups were.
The Unknowns: They hoped to find new, undiscovered groups.

The Results:

Old Methods: The "Strict Librarian" (C-DBSCAN) got confused. It saw a bridge of stars connecting two different ancient groups and merged them into one giant, messy blob. It missed the new discoveries entirely.
CLiMB:
1. Recovered the Knowns: It perfectly identified the 8 known star groups (90% accuracy).
2. Found the New: In the "leftover" data, it found three distinct new groups:
  - The Galactic Disk (the main flat part of the galaxy).
  - Shiva and Shakti (two mysterious, newly confirmed ancient star groups that were previously hidden in the noise).

Why This Matters

CLiMB is like a detective who is smart enough to know exactly what a "fingerprint" looks like (so they don't get confused by similar-looking marks) but is also curious enough to investigate the strange smudges on the wall that don't match any known fingerprint.

By separating the job of "Confirming what we know" from "Exploring what we don't," CLiMB allows scientists to clean up their data and find hidden cosmic secrets that other methods simply sweep under the rug.

1. Problem Statement

In data-driven scientific discovery, particularly in Galactic archaeology, researchers face a dual challenge:

Recovering Known Structures: Accurately classifying well-characterized phenomena (e.g., known stellar streams like Gaia Sausage/Enceladus) using existing domain knowledge.
Detecting Novelty: Identifying unanticipated anomalies or new structures within the same dataset without being biased by prior assumptions.

Limitations of Current Methods:

Unsupervised Clustering: Treats all data as unknown, ignoring vast amounts of accumulated domain knowledge, leading to poor recovery of known substructures.
Standard Semi-Supervised Clustering (SSC): Often enforces rigid global constraints (e.g., must-link/cannot-link) or assumes supervisory signals are globally representative. This can suppress novel patterns, cause "chain-reaction" merging of distinct structures in high-density regions, or fail to discover new topologies.
Metric Learning: While powerful, distance metrics learned from known classes often lack "local validity" when applied to the geometry of unknown regions.

2. Methodology: The CLiMB Framework

The authors propose CLiMB (CLustering in Multiphase Boundaries), a domain-informed framework that decouples the exploitation of prior knowledge from the exploration of unknown structures using a sequential two-phase architecture.

Phase 1: K-Bound (Constrained Clustering)

This phase anchors known clusters using user-provided "seed points" (prior knowledge) while strictly enforcing boundaries to prevent "semantic drift."

Algorithm: A revised K-means variant.
Three Key Constraints:
1. Density Constraint: Points are only assigned if their local density (estimated via Gaussian Kernel Density Estimation) exceeds a threshold ( $\tau_\rho$ ). Low-density points are deferred to Phase 2.
2. Distance Constraint: Assignment requires the point to be within a maximum distance threshold ( $\tau_d$ ) of the nearest centroid, ensuring cluster compactness.
3. Radial Constraint: Centroids are constrained to remain within a specific radius ( $\tau_r$ ) of their initial seed positions. This prevents centroids from drifting too far due to noise, ensuring the final clusters remain faithful to the prior knowledge.
Metric Flexibility: Supports Euclidean, Mahalanobis (critical for handling correlations in kinematic data like Energy and Angular Momentum), and custom metrics.
Initialization: Uses seed points to define initial centroids, guaranteeing that known structures are respected.

Phase 2: Exploratory Novelty Detection

This phase processes the residual data (points unassigned in Phase 1) to discover new structures.

Approach: Applies density-based clustering algorithms (default: DBSCAN; also supports HDBSCAN, OPTICS).
Goal: To reveal arbitrary, non-convex topologies without forcing them into the mold of the priors.
Output: New cluster labels are indexed distinctly from the constrained clusters.

3. Key Contributions

Architectural Decoupling: CLiMB formally separates "constraint satisfaction" (exploitation) from "density-based exploration," resolving the tension between adhering to prior knowledge and discovering new structures.
Domain-Informed Design: Specifically tailored for Galactic Archaeology, utilizing the Integrals of Motion (IoM) space (Total Energy $E$ , Angular Momentum $L_z$ , Perpendicular Angular Momentum $L_\perp$ ) and Mahalanobis distances to handle the elongated, tilted shapes of stellar streams.
Robustness to Sparsity: The framework is designed to handle the high background noise and varying densities typical of Gaia data.
Open Source Implementation: The algorithm is implemented in Python, compatible with scikit-learn, and publicly available on GitHub.

4. Experimental Results

The framework was evaluated on 4,933 RR Lyrae stars from the Gaia Data Release 3 (DR3).

Performance Metrics:

Knowledge Recovery: CLiMB achieved an Adjusted Rand Index (ARI) of 0.829 (with 90% seed coverage) in recovering 8 known Milky Way substructures.
- Comparison: Outperformed C-DBSCAN (ARI: 0.152) and Heuristic SS-DBSCAN (ARI: 0.040).
- Analysis: Baselines suffered from "chain-reaction" merging or fragmentation, whereas CLiMB's radial constraints prevented distinct streams from merging.
Novelty Discovery: In the unlabelled residual data, CLiMB successfully isolated three distinct dynamical features:
1. The Galactic Disk.
2. The Shiva substructure.
3. The Shakti substructure.
- These findings were validated against recent astrophysical literature (Malhan and Rix, 2024).
Sensitivity Analysis: CLiMB demonstrated monotonic improvement in performance as the percentage of seed data increased (from 10% to 100%). In contrast, baseline methods stagnated regardless of supervision levels. Even with only 10% seeds, CLiMB's ARI (~0.32) doubled that of the best baseline.

5. Significance and Impact

Scientific Discovery: CLiMB moves beyond mere classification to act as a hypothesis generation tool. By effectively "cleaning" the feature space of known structures in Phase 1, it allows Phase 2 to isolate subtle, novel signals (like Shiva and Shakti) that are typically obscured by background noise in standard clustering.
Methodological Advancement: It addresses a critical gap in semi-supervised learning where existing methods fail to balance the rigidity of constraints with the flexibility required for novelty detection.
Applicability: While demonstrated in astronomy, the framework is applicable to any domain where well-characterized classes coexist with unknown anomalies (e.g., bioinformatics, fraud detection).

6. Limitations and Future Work

Hyperparameter Complexity: The two-phase architecture requires tuning more parameters (density, distance, and radial thresholds) compared to monolithic algorithms.
Computational Cost: The initial Gaussian Kernel Density Estimation has a complexity of $O(n^2d)$ , which can be a bottleneck for datasets exceeding millions of points. Future work suggests integrating approximate nearest-neighbor structures (e.g., kd-trees).
Sequential Dependency: The quality of novelty detection is strictly dependent on the success of the first phase; overly strict constraints could leak known points into the residual set, contaminating the discovery process.