Unpaired Image-to-Image Translation via a Self-Supervised Semantic Bridge

This paper proposes the Self-Supervised Semantic Bridge (SSB), a framework that leverages self-supervised visual encoders to create a shared geometric latent space for diffusion bridge models, enabling high-fidelity, spatially faithful unpaired image-to-image translation without cross-domain supervision.

The Gist

Imagine you are a master translator, but instead of translating words between French and English, you are translating images between two completely different worlds.

Let's say you want to turn a MRI scan (which looks like a fuzzy, gray ghost of a body) into a CT scan (which looks like a sharp, high-contrast X-ray). Or, you want to turn a photo of a horse into a zebra, or a summer landscape into a winter wonderland.

The problem? You don't have a "dictionary" that pairs every single MRI with its matching CT scan. You have a pile of MRIs and a separate pile of CTs, but they aren't lined up. This is called Unpaired Image-to-Image Translation.

For a long time, computers struggled with this. They either got the shapes wrong (the horse's legs turned into zebra stripes but the legs disappeared) or they got the colors right but the anatomy was a mess.

Enter the new method from this paper: SSB (Self-Supervised Semantic Bridge).

Here is how it works, using a simple analogy:

The Problem: The "Lost in Translation" Effect

Imagine you are trying to translate a book from a language you don't speak (Source) to another language you don't speak (Target), and you have no dictionary.

• Old Method A (The Adversarial Approach): You hire two people. One tries to fake a translation, and the other tries to catch them lying. They argue back and forth until they agree on a translation. But if the "fake" translator gets too creative, the story loses its original meaning.
• Old Method B (The Inversion Approach): You try to turn the book into pure noise (static) and then try to rebuild it in the new language. But because the "noise" is messy, you often lose the plot details. The story comes back, but the characters are in the wrong places.

The Solution: The "Universal Blueprint" (The Semantic Bridge)

The authors of this paper realized that while the appearance of an MRI and a CT scan are totally different (gray vs. white, fuzzy vs. sharp), the skeleton underneath is exactly the same. The liver is in the same spot; the spine curves the same way.

They built a Semantic Bridge. Think of this bridge as a Universal Blueprint or a 3D Wireframe.

1. The Smart Architect (The Encoder):
   They used a special AI (called DINO) that acts like a super-architect. This architect doesn't care about the paint color, the wallpaper, or the lighting. It only cares about the structure.

   • If you show it a photo of a horse, it ignores the brown fur and sees the "Horse Shape."
   • If you show it a photo of a zebra, it ignores the black-and-white stripes and sees the "Horse Shape."
   • If you show it an MRI, it ignores the gray fuzz and sees the "Body Shape."
   • If you show it a CT, it ignores the white bones and sees the "Body Shape."

   This architect creates a Shared Blueprint (a "latent space") that is identical for both the horse and the zebra, or the MRI and the CT.

2. The Construction Crew (The Diffusion Bridge):
   Once the blueprint is made, the computer doesn't need to guess how to turn an MRI into a CT. It just needs to say: "Here is the blueprint. Now, build a CT scan that fits this blueprint."

   • It takes the MRI, extracts the blueprint.
   • It uses that blueprint to guide a "construction crew" (a diffusion model) to build a brand new CT scan from scratch, ensuring the bones and organs are in the exact right places.

Why is this a Big Deal?

1. No "Pairing" Required:
You don't need to find a patient who has both an MRI and a CT scan taken at the exact same time. You can just throw a million MRIs and a million CTs into the system, and the "Architect" will figure out the shared shapes on its own. This is like learning to translate French and English just by reading a million French books and a million English books, without ever seeing a sentence that is written in both.

2. It Works on "Weird" Data (Out-of-Distribution):
Imagine you trained your translator on French books about cats, but then someone hands you a French book about space travel. Old methods would get confused and output gibberish.
Because SSB focuses on the structure (the grammar and logic) rather than just the specific words (the appearance), it can handle new types of MRI scans it has never seen before. It knows that "a kidney looks like a kidney" even if the contrast is weird.

3. It's a One-Way Street (Scalability):
If you want to add a third language (say, Ultrasound), you don't need to pair Ultrasound with MRI and Ultrasound with CT. You just teach the Architect how to read Ultrasound blueprints. Now, the bridge connects everything automatically. It's much cheaper and faster to scale.

Real-World Magic

The paper shows this working on:

• Medical Imaging: Turning fuzzy MRIs into sharp CTs so doctors can plan surgery without needing a CT scan (which uses radiation).
• Art & Editing: Turning a summer photo into a winter one, or changing a horse into a zebra, while keeping the horse's pose and the background perfectly intact.

The Catch (Limitations)

The system is great at changing the "skin" (color, texture, style) while keeping the "bones" (shape, structure) the same.

• It struggles with big changes: If you ask it to turn a small lizard into a giant dragon, it gets confused. The "blueprint" says "lizard," but the prompt says "dragon." The system tries to keep the lizard shape, so the result might look like a weird, stretched lizard-dragon hybrid. It's too stubborn to completely change the geometry.
• Abstract inputs: If you try to translate a stick-figure drawing into a photo, it might fail because the "blueprint" isn't detailed enough to start with.

The Bottom Line

SSB is like a master translator who ignores the accent and the vocabulary and focuses entirely on the story's plot. By building a shared "structural blueprint" between two different worlds, it allows computers to translate images with incredible accuracy, even when they've never seen the two images paired together before. It's a huge step forward for medical imaging and creative AI editing.

Technical Summary

1. Problem Statement

Unpaired image-to-image (I2I) translation aims to transfer semantic content between distinct domains (e.g., MRI to CT, Horse to Zebra) without paired training data. Existing approaches face two primary limitations:

• Adversarial Methods (GANs): Rely on cross-domain adversarial losses and cycle-consistency constraints. These often struggle with generalization to out-of-domain (OOD) data (e.g., unseen MRI contrasts) and can suffer from mode collapse or instability.
• Diffusion-Inversion Methods: Translate images by inverting them into the noise latent space of a pre-trained diffusion model and re-synthesizing them. However, inversion is often imperfect, leading to error propagation, structural drift, and low-fidelity translations. Furthermore, many rely on injecting intermediate features (like attention maps) tied to specific architectures, limiting transferability.

The core challenge is to achieve spatially faithful translation that preserves geometric structure while adapting appearance, without requiring paired data or explicit cross-domain alignment objectives.

2. Methodology: Self-Supervised Semantic Bridge (SSB)

The authors propose SSB, a framework that bridges source and target domains via a shared semantic latent space derived from self-supervised visual encoders, rather than direct domain coupling.

Core Components

1. Shared Semantic Latent Space:

   • The method assumes that images from different domains share a common latent representation $y$ that captures geometric structure but is invariant to appearance changes.
   • Instead of learning a direct mapping $x_{source} \to x_{target}$, SSB maps $x_{source} \to y$ and then $y \to x_{target}$.
   • Encoder ($E_\phi$): A self-supervised visual encoder (based on DINO/DINOv2) is used to extract patch embeddings. These embeddings are naturally invariant to appearance changes (color, contrast) but preserve geometric structure.
   • Projection: The encoder outputs are projected (via PCA) to match the dimensionality of the target domain's VAE latent space.

2. Diffusion Bridge Models:

   • SSB utilizes Diffusion Bridges (or Stochastic Interpolants) to learn the path between the shared latent $y$ and the target domain's latent distribution.
   • Training: Each domain trains its own bridge model independently to connect its specific latent space to the shared endpoint $y$. There is no cross-domain supervision or adversarial loss during training.
   • Inference:
     1. Encode the source image $x^{(j)}$ into the shared latent $y = E_\phi(x^{(j)})$.
     2. Sample from the target domain's bridge model conditioned on $y$ to generate the target latent $\bar{z}^{(i)}$.
     3. Decode $\bar{z}^{(i)}$ to obtain the final image $\bar{x}^{(i)}$.

3. Handling Ambiguity (Deterministic vs. Stochastic Endpoints):

   • For tasks with strong geometric correspondence (e.g., MRI $\to$ CT), the endpoint is set deterministically ($b=0$), strictly preserving fidelity.
   • For tasks with high appearance ambiguity (e.g., natural images), a stochastic endpoint ($b>0$) is used, allowing the diffusion process to synthesize valid domain-specific details while maintaining the geometric center.

4. Appearance-Invariant Encoder Training:

   • To ensure the encoder truly ignores appearance, the authors fine-tune DINOv2 with a retina-inspired filter. This filter suppresses low-frequency appearance cues (like contrast differences between MRI and CT) and enhances high-frequency structural boundaries, forcing the model to learn geometry-consistent representations.

5. Vector Field Interpolation:

   • To balance structural preservation and appearance adaptation during generation, the method interpolates between the source and target drift fields using a time-varying coefficient $\eta_t$. This allows for smooth transitions where structural consistency is prioritized early in the generation process, with appearance modulation relaxing constraints later.

3. Key Contributions

• Novel Framework: Introduction of SSB, the first unpaired I2I framework that decouples translation from cross-domain training by using a self-supervised shared semantic manifold. This enables linear scaling to new domains (training only one new single-domain model) rather than pairwise coupling.
• Theoretical Justification: A theoretical error analysis (Theorem 4.1) bounding the translation error based on encoder alignment, vector field approximation, and discretization errors, proving that the method remains robust even with imperfect encoder alignment.
• Medical Imaging Breakthrough: Development of a geometry-aware MRI-CT representation via DINOv2 fine-tuning, achieving performance comparable to supervised methods in both in-domain and challenging out-of-domain (OOD) settings (e.g., unseen MRI contrasts).
• Versatility: Successful extension to natural image translation (Horse $\to$ Zebra) and text-guided editing (Scene and Object editing) using Stable Diffusion 3, demonstrating broad applicability.

4. Experimental Results

The paper evaluates SSB against strong baselines including CycleGAN, UNIT, SDEdit, DDIB, and recent diffusion bridge methods (I2SB, SelfRDB).

• Medical Image Synthesis (MRI $\to$ CT):
  • In-Domain: SSB achieves state-of-the-art (SOTA) performance with an FID of 30.15 (vs. 42.75 for DDBM) and high MS-SSIM (0.810).
  • Out-of-Domain (OOD): SSB significantly outperforms all baselines on unseen MRI contrasts (UKBB dataset), achieving an MS-SSIM of 0.585 compared to 0.540 for the next best method (DDBM). This highlights superior generalization.
• Natural Image Translation:
  • On Horse $\to$ Zebra and Apple $\to$Orange, SSB achieves the best balance between text adherence (CLIP-T) and structural preservation (DINO similarity, PSNR), outperforming CycleGAN and ControlNet.
• Text-Guided Editing:
  • Using SD3-M, SSB provides competitive or superior results in scene-style transfer and object-level editing, offering a better trade-off between following the text prompt and preserving the source image structure compared to FlowEdit and ControlNet.
• Efficiency: SSB demonstrates competitive inference speeds (e.g., 2.57s/image for MRI $\to$ CT) compared to other diffusion-based methods.

5. Significance and Impact

• Decoupling Domains: SSB fundamentally changes the paradigm of unpaired translation by removing the need for explicit domain coupling or adversarial training. This makes the approach highly scalable and robust to distribution shifts (OOD).
• Medical Applications: The ability to synthesize CT scans from MRI scans without paired data and with robustness to different MRI contrasts is critical for radiotherapy planning and reducing patient radiation exposure.
• Structure Preservation: By leveraging self-supervised geometric priors, SSB solves the "structural drift" problem common in inversion-based diffusion methods, ensuring that anatomical or object structures remain intact during translation.
• Generalizability: The framework is not limited to medical imaging; it provides a unified solution for diverse I2I tasks, from style transfer to text-guided editing, establishing a new baseline for structure-preserving generative modeling.

In summary, SSB represents a significant advancement in unpaired image translation by integrating self-supervised semantic priors into diffusion bridges, achieving high-fidelity, structurally consistent, and generalizable results across both medical and natural domains.