Learning What's Real: Disentangling Signal and Measurement Artifacts in Multi-Sensor Data, with Applications to Astrophysics

This paper proposes a deep learning framework that disentangles intrinsic physical signals from sensor-specific artifacts in multi-instrument data by leveraging overlapping observations and counterfactual generation, thereby enabling unconfounded parameter inference and instrument-independent analysis, as demonstrated on astrophysical galaxy images.

The Gist

Imagine you are trying to listen to a beautiful song played on a violin. But there's a problem: the room you are in is echoing, the microphone is slightly distorted, and there's a hum from the air conditioner.

If you just record the sound, you get a messy mix of the violin (the real signal) and the room/microphone (the noise and artifacts). In science, this is a huge problem. Astronomers look at the universe through telescopes, but every telescope has its own "personality." One might make stars look blurry, another might add a weird color tint, and a third might be very sensitive to noise.

For a long time, scientists had to manually try to "clean" these images, like trying to remove the echo from a recording by guessing what the echo sounded like. It was slow, difficult, and often imperfect.

This paper introduces a new, smart way to solve this using Artificial Intelligence. Here is how it works, explained simply:

1. The Problem: The "Bad Room" vs. The "Real Song"

The authors call the real thing the Physics (the galaxy, the star, the sound) and the messiness the Instrument (the telescope, the camera, the microphone).

• The Goal: They want to teach a computer to separate the "song" from the "room noise" automatically.
• The Challenge: Usually, you only have one recording. How do you know what the noise sounds like if you don't know what the clean song sounds like?

2. The Solution: The "Time-Traveling" Trick

The secret sauce of this paper is using overlapping observations. Imagine you have a photo of the same galaxy taken by two different telescopes:

• Telescope A (The "Legacy" Survey): Takes a wide view of the sky but the images are a bit fuzzy and low-resolution.
• Telescope B (The "HSC" Survey): Takes a very sharp, high-resolution view but only of a tiny patch of sky.

Because they both looked at the same galaxy, the AI can learn a powerful trick:

• It looks at the fuzzy image and the sharp image.
• It realizes: "Ah, the shape of the galaxy is the same in both, but the sharpness and the grainy noise are different."

3. The AI Architecture: The "Dual-Brain" System

The researchers built a special AI with two "brains" (encoders) and a "reconstruction artist" (decoder):

• Brain 1 (The Physics Detective): This brain looks at the galaxy through the "wrong" telescope. Its job is to ignore the telescope's quirks and only learn the true shape and color of the galaxy. It asks, "What does this galaxy really look like, regardless of which camera took the picture?"
• Brain 2 (The Instrument Detective): This brain looks at a different galaxy taken by the same telescope. Its job is to ignore the galaxy's shape and only learn the camera's quirks (the blur, the noise, the color tint). It asks, "What does this specific camera do to any picture?"
• The Reconstruction Artist (The Decoder): This part takes the "True Shape" from Brain 1 and the "Camera Quirks" from Brain 2 and tries to paint a picture.

4. The Magic: "Counterfactual" Generation

Here is the coolest part. The AI is trained using a "what if" game (Counterfactuals).

• The Game: The AI is shown a galaxy taken by Telescope A. It is not allowed to see the original Telescope A photo while it learns. Instead, it has to guess what that galaxy would look like if it were taken by Telescope B.
• The Result: The AI learns to strip away Telescope A's noise and add Telescope B's style. It essentially says, "If I took this fuzzy picture and ran it through the sharp camera, here is what it would look like."

Because the AI has to do this perfectly to win the game, it is forced to learn exactly what is "real" (the galaxy) and what is "fake" (the telescope noise).

5. Why This Matters (The Real-World Impact)

The authors tested this on over 100,000 galaxy images. Here is what they found:

• Super-Resolution: They can take a fuzzy, low-quality image from a wide survey and "hallucinate" (generate) what it would look like if taken by a super-powerful, expensive telescope. This helps astronomers find rare objects (like gravitational lenses) without needing to point the expensive telescope at every single star.
• Fair Comparisons: Now, scientists can compare galaxies from different telescopes as if they were all taken by the same camera. It removes the bias.
• The "Universal Translator": The AI creates a "clean" language of galaxies. Whether you speak "Telescope A" or "Telescope B," the AI translates both into the same pure language of physics.

The Analogy Summary

Think of it like noise-canceling headphones, but instead of canceling sound, it cancels camera distortion.

• Old Way: You try to manually fix the photo in Photoshop, guessing where the blur came from.
• New Way: You show the AI a photo taken in a noisy room and a photo of the same person taken in a quiet studio. The AI learns the "noise" of the room and the "face" of the person separately. Then, it can take a new photo of that person in a noisy room and instantly show you what they would look like in the quiet studio.

This framework allows scientists to see the universe more clearly, separating the truth of the cosmos from the limitations of our tools.

Technical Summary

1. Problem Statement

Observational data in science is inherently a mixture of two components:

1. Intrinsic Signal: The true physical phenomenon of interest (e.g., the light from a distant galaxy).
2. Measurement Artifacts: Distortions, noise, and biases introduced by the specific sensor or instrument (e.g., Point Spread Function variations, detector non-linearity, atmospheric effects).

The Challenge:

• Confounding: Instrument effects act as confounding factors, making it difficult to extract pure physical information.
• Heterogeneity: Combining data from different instruments (multi-instrument settings) is difficult because the same physical object appears differently due to varying resolutions, noise profiles, and calibration.
• Limitations of Existing Methods:
  • Contrastive Learning: Often requires carefully designed positive/negative pairs and struggles to generate counterfactual views (e.g., "what would this look like on a different telescope?").
  • Explicit Modeling: Requires detailed, often intractable analytical models of instrument physics.
  • Standard Foundation Models: Often treat different instruments as independent modalities without explicitly disentangling the instrument-specific noise from the physics, leading to latent spaces dominated by systematics rather than physics.

2. Methodology

The authors propose a Counterfactual Generative Framework using a dual-encoder architecture and Conditional Flow Matching.

Core Concept: Counterfactual Generation

The model learns to reconstruct an observation from a target instrument by conditioning on:

1. Physics Latents ($Z_{phy}$): Derived from the same source observed by different instruments.
2. Instrument Latents ($Z_{ins}$): Derived from different sources observed by the same target instrument.

The goal is to learn the conditional distribution:
$$p(x_{j,k} \mid \{z_{phy}(x_{j,k'})\}_{k' \neq k}, \{z_{ins}(x_{j',k})\}_{j' \neq j})$$
Where $x_{j,k}$ is the observation of source $j$ by instrument $k$.

Architecture

• Dual Encoders:
  • Physics Encoder: Takes an image of the target source from a different instrument. It is trained to be invariant to instrument distortions, capturing only intrinsic properties.
  • Instrument Encoder: Takes an image of a different source from the target instrument. It is trained to capture instrument-specific systematics (noise, PSF, depth) while ignoring the source content.
• Generative Decoder: A UNet based on Flow Matching (Lipman et al., 2023).
  • It learns a velocity field $u_\theta$ that transports a Gaussian prior to the target data distribution.
  • It is conditioned on the concatenated physics and instrument latent embeddings via cross-attention.
• Training Objective: The model minimizes the error between the predicted velocity field and the true velocity field required to reconstruct the Anchor Image (the ground truth observation of the source on the target instrument).
  • Crucial Detail: The anchor image is never fed into the encoders. It is only used as the target for the flow-matching loss. This forces the model to generate the image "counterfactually" based solely on the disentangled latents.

Data Strategy (Triplets)

The training data consists of triplets constructed from overlapping observations:

1. Anchor: Source $S$ observed by Instrument $A$.
2. Physics View: Source $S$ observed by Instrument $B$ (provides $Z_{phy}$).
3. Instrument View: Source $S'$ (different source) observed by Instrument $A$ (provides $Z_{ins}$).

3. Key Contributions

1. Architecture-Driven Disentanglement: Unlike previous methods relying on hand-engineered contrastive losses, this framework enforces structural disentanglement through the architecture (dual encoders) and the generative objective itself.
2. Counterfactual Generation: The model can synthesize how a source would appear under different observing conditions (e.g., simulating high-resolution HSC images from low-resolution Legacy Survey data).
3. Black-Box Instrument Modeling: The framework treats instruments as black boxes, learning their effects implicitly from data without requiring explicit physical models of the optics or detectors.
4. General Recipe for Science: The authors propose a general self-supervised pretraining recipe: construct pairs from overlapping observations, treat instrument effects as augmentations, and learn invariant representations via counterfactual generation.

4. Experimental Results

The method was validated on ~100,000 cross-matched galaxy images from the DESI Legacy Imaging Survey (Legacy) and the Hyper Suprime-Cam (HSC) Survey.

• Reconstruction Quality:
  • The model successfully reconstructs target images with low Mean Squared Error (MSE: 0.081 for HSC-anchored, 0.197 for Legacy-anchored).
  • Generated posterior samples capture realistic uncertainty and noise characteristics.
• Uncertainty Calibration:
  • Pixel-wise Z-scores of generated samples follow a standard normal distribution, indicating well-calibrated uncertainty estimates.
• Latent Space Disentanglement (UMAP Visualization):
  • Instrument Space: HSC and Legacy images form distinct, non-overlapping clusters, proving the encoder captures survey-specific characteristics.
  • Physics Space: HSC and Legacy images of the same galaxy map to overlapping regions, confirming the extraction of domain-invariant physical features.
• Downstream Task Performance:
  • Physics Prediction: The physics latent space predicts physical properties (redshift, stellar mass, morphology) with accuracy comparable to monolithic foundation models (like AION-1) but is robust to instrument systematics.
  • Instrument Prediction: The instrument latent space outperforms baselines in predicting instrument-specific properties (PSF size, depth).
  • Cross-Survey Transfer: A model trained on real HSC data to predict galaxy ellipticity achieves $R^2 = 0.81$ on counterfactual HSC images generated from Legacy data (vs. $R^2 = 0.82$ on real HSC). This allows existing HSC pipelines to be applied to Legacy data with minimal loss of accuracy.
• Similarity Search:
  • Physics Space: Enables instrument-independent search (finding physically similar galaxies regardless of which telescope took the image).
  • Instrument Space: Enables search for objects with similar noise/systematic characteristics.

5. Significance and Future Impact

• Scientific Efficiency: The framework acts as a "learned survey simulator," enabling astronomers to prioritize follow-up observations on expensive, high-resolution telescopes (like JWST) by first generating counterfactuals from cheaper, wide-field surveys (like Legacy).
• Data Integration: It provides a robust method for merging heterogeneous datasets, a critical need as multi-messenger and multi-survey astronomy expands.
• Scalability: The approach is designed to scale to settings with more than two instruments and is applicable beyond astrophysics (e.g., medical imaging, remote sensing) where multiple sensors observe the same physical phenomena.
• Limitations & Future Work: Currently requires overlapping sky coverage (paired data). Future work aims to extend this to unpaired settings and learn per-instrument residual latents to capture high-resolution details unique to specific instruments that are lost in the shared physics representation.

In summary, this paper presents a powerful generative framework that fundamentally separates "what is real" (physics) from "how we see it" (instrument), enabling more accurate scientific inference and flexible data synthesis across diverse observational platforms.