A Multi-modal Fusion Network for Star-Galaxy Classification from CSST Simulated Datasets

This paper proposes a multi-modal deep learning network combining ResNet-50 and BiLSTM to fuse image and catalog data from China Space Station Telescope (CSST) simulations, achieving over 99.6% recall for star-galaxy classification and demonstrating robust performance on faint and high-redshift objects.

The Gist

Imagine you are a librarian trying to organize a massive, chaotic library that is about to explode with new books. In fact, this library is the universe, and the "books" are billions of stars and galaxies. For centuries, astronomers had to manually look at each one to tell them apart, but with new telescopes coming online (like the China Space Station Telescope, or CSST), they will be flooded with so much data that human eyes simply can't keep up.

This paper is about building a super-smart robot librarian that can instantly and accurately sort these cosmic books into two piles: "Stars" and "Galaxies."

Here is how they built this robot, explained simply:

1. The Problem: The "Point" vs. The "Blob"

In the night sky, stars usually look like tiny, sharp pinpricks of light (like a needle point). Galaxies, on the other hand, look like fuzzy, glowing clouds or smudges (like a cotton ball).

• The Old Way: Astronomers used to look at a single photo or a list of numbers (how bright the object is in different colors) to guess what it was. Sometimes, a faint galaxy looks so small and tight that it tricks the eye into thinking it's a star.
• The New Challenge: The CSST telescope will take pictures in seven different colors (from ultraviolet to infrared) and also generate a detailed "ID card" (catalog) for every object. The challenge is to combine all this information without getting confused.

2. The Solution: The "Two-Brain" Robot (RBiM Network)

The researchers created a deep learning model they call RBiM. Think of this model as a detective with two specialized brains working together:

• Brain A (The Artist - ResNet-50): This brain looks at the pictures. It analyzes the seven different colored photos of the object. It's like an artist looking at a painting to see if the brushstrokes are sharp (a star) or fuzzy (a galaxy). It uses a special "attention mechanism" to zoom in on the most important details, ignoring the background noise.
• Brain B (The Accountant - BiLSTM): This brain looks at the numbers (the catalog). It reads the "ID card" which lists how bright the object is in each of the seven colors. It's like an accountant checking a budget sheet to see if the spending pattern matches a star or a galaxy. It looks at the data from left to right and right to left to understand the full story.

The Magic Fusion:
Usually, you might ask the Artist and the Accountant for their opinions separately and then take a vote. But this robot is smarter. It fuses their brains. It combines the visual details from the picture with the numerical patterns from the ID card before making a final decision. This is like the Artist and Accountant sitting at the same table, pointing at the same evidence, and agreeing on the answer together.

3. Training the Robot

To teach this robot, the researchers didn't use real telescope data yet (because the telescope isn't fully active). Instead, they used a super-realistic video game simulation of the universe.

• They created a dataset with about 32,000 stars and 93,000 galaxies.
• The Imbalance Problem: There were way more galaxies than stars. If you train a robot on this, it gets lazy and just guesses "Galaxy" for everything to get a high score.
• The Fix: They used Data Augmentation. Imagine taking a photo of a star, flipping it upside down, rotating it, and mirroring it. They did this to create three times as many "fake" star photos. This forced the robot to actually learn what a star looks like, rather than just guessing.

4. The Results: A Masterpiece of Sorting

After training for a while, the robot was tested, and the results were incredible:

• Accuracy: It got it right 99.8% of the time.
• The "Faint" Test: The hardest objects to sort are the dim, distant ones (like looking at a candle from a mile away). Traditional methods often fail here, confusing dim galaxies for stars. This robot, however, kept its cool, maintaining high accuracy even for very faint objects.
• The "Missing Data" Test: What if the robot only gets a picture in blue light but not red light? Even with missing information, it still performed brilliantly, showing it's very robust.

5. Why This Matters

Think of the upcoming CSST telescope as a floodgate opening. It will pour out terabytes of data. If we try to sort this manually, we'll drown.
This paper proves that by combining images (what it looks like) and data (how bright it is in different colors) using a smart AI, we can build a system that is:

1. Fast: It can process millions of objects instantly.
2. Accurate: It rarely makes mistakes, even on the tricky, faint objects.
3. Ready for the Future: It is specifically designed to handle the massive data deluge that the China Space Station Telescope will bring.

In short, the researchers built a super-sorted, multi-sensory robot that will help astronomers clean up the universe's library, ensuring that when they study the cosmos, they are looking at the right kind of "books."

Technical Summary

1. Problem Statement

The distinction between stars and galaxies is a fundamental challenge in observational astronomy, critical for understanding cosmic structure and evolution. With the advent of massive digital surveys like the China Space Station Telescope (CSST), which will generate terabytes to petabytes of data containing billions of objects, traditional manual classification and single-modality methods (relying solely on photometric catalogs or single-band images) are insufficient.

• Limitations of Current Methods: Existing approaches often rely on catalog data (magnitudes and imprecise morphological parameters) or single-band images. These methods struggle with faint objects, high-redshift galaxies, and scenarios where morphological features are obscured by noise or the Point Spread Function (PSF).
• Data Imbalance: In deep surveys, galaxies often outnumber stars, leading to classification bias where models favor the majority class to maximize accuracy, failing to learn substantive features of minority classes.

2. Methodology

The authors propose a Multi-modal Fusion Network (RBiM) that integrates deep learning techniques to process both multi-band images and photometric catalogs simultaneously.

A. Dataset Construction

• Source: CSST simulation data covering a $0.75 \text{ deg}^2$ region with 7 photometric bands (NUV, u, g, r, i, z, y).
• Composition: 32,371 stars and 93,525 galaxies.
• Preprocessing:
  • Cross-matching: Input catalogs and photometric catalogs were matched with a positional error tolerance of 0.3 arcseconds.
  • Image Cutouts: Mosaic images were cropped into $50 \times 50$ pixel cutouts centered on each source, stacked into 7-channel FITS images.
  • Data Augmentation: To address class imbalance, star samples were augmented 3-fold using random horizontal/vertical flips and rotations ($\pm 45^\circ$). Catalog data was copied to maintain image-catalog correspondence.

B. Network Architecture (RBiM)

The model combines ResNet-50 and Bi-directional Long Short-Term Memory (BiLSTM) within a PyTorch framework.

1. Image Branch (ResNet-50 + Attention):
   • Uses a 5-stage ResNet-50 backbone to extract morphological features from the 7-channel images.
   • Attention Mechanism: Integrated progressively after each stage, comprising Channel Attention (to capture dependencies across the 7 bands) and Spatial Attention (to highlight significant regions in the feature map).
2. Catalog Branch (BiLSTM):
   • Processes the 7-band magnitude data (Spectral Energy Distribution - SED).
   • Uses two independent LSTM layers (forward and backward) to capture causal dependencies between bands (e.g., NUV to $y$ and $y$ to NUV), effectively modeling the SED even with missing bands.
3. Fusion Layer:
   • Features from the ResNet-50 (flattened) and BiLSTM are concatenated.
   • The fused vector is passed through a Multi-Layer Perceptron (MLP) with three layers to output classification probabilities.

3. Key Contributions

• Multi-modal Fusion: Unlike previous decision-level fusion methods, this approach performs feature-level fusion, integrating deep visual features with photometric sequence data to leverage complementary information.
• Robustness to Missing Data: The BiLSTM architecture allows the model to handle incomplete catalogs (missing red or blue bands) effectively, a common issue in real-world survey data.
• Handling Faint and High-Redshift Objects: The model specifically addresses the difficulty of classifying compact, faint galaxies and high-redshift objects where morphological features are indistinguishable from stars.
• Comprehensive Evaluation: The study includes rigorous ablation studies comparing the fusion model against "catalog-only" and "image-only" baselines, as well as traditional SExtractor methods.

4. Experimental Results

The model was trained for 50 epochs on GPU using 3-fold cross-validation, label smoothing, and cosine annealing learning rate scheduling.

• Overall Performance:

  • Galaxies: 99.81% Recall, 99.88% Precision.
  • Stars: 99.66% Recall, 99.44% Precision.
  • Overall Accuracy: >99.75%.
  • Error Analysis: Only 18 galaxies were misclassified as stars, and 11 stars as galaxies in the test set (12,590 total samples).

• Ablation Studies:

  • Multi-modal Advantage: Compared to a catalog-only model, the fusion model improved galaxy recall by 11.17% and star precision by 25.01%. Compared to image-only, it showed marginal but consistent gains, proving the value of catalog data.
  • Data Augmentation: Removing augmentation reduced recall by 0.05% (galaxies) and 0.16% (stars), confirming its necessity for handling class imbalance.

• Robustness Tests:

  • Missing Bands: The model maintained >98% accuracy even when red (z, y) or blue (NUV, u) bands were missing.
  • Magnitude Sensitivity: For objects fainter than 23 mag, the fusion model maintained a low misclassification rate, whereas the traditional SExtractor "spread model" error rate rose to 30% at 26 mag.
  • Redshift Sensitivity: For high-redshift galaxies ($z=1$ to $2$), the fusion model maintained a misclassification rate of <0.5%, while the spread model error rate exceeded 20%.

5. Significance and Conclusion

This paper demonstrates that a multi-modal deep learning approach significantly outperforms traditional single-modality and rule-based methods (like SExtractor) for star-galaxy classification.

• Scientific Impact: The model is particularly effective for faint sources and high-redshift galaxies, which are the most challenging yet scientifically valuable targets for the upcoming CSST survey.
• Future Applicability: The authors conclude that the RBiM network is highly adaptable. With minimal fine-tuning via transfer learning, it can be applied to real CSST measured data, despite the increased noise and complexity compared to simulations.
• Methodological Advance: The successful integration of CNNs (for spatial morphology) and BiLSTMs (for spectral sequence data) provides a robust framework for future multi-modal astronomical classification tasks.