Bridging Day and Night: Target-Class Hallucination Suppression in Unpaired Image Translation

This paper proposes a novel framework for day-to-night unpaired image translation that utilizes a dual-head discriminator to detect semantic hallucinations and class-specific prototypes within a Schrödinger Bridge-based model to suppress these artifacts, thereby significantly improving downstream object detection performance.

The Gist

Imagine you are a master chef trying to teach a robot how to cook a "Nighttime Dinner" using only recipes for "Daytime Lunch." The robot has never seen a night scene, so it has to guess what things look like when the sun goes down.

The problem? The robot gets creative in the wrong ways. It sees a street sign in the daytime photo and thinks, "Ah, at night, this must be a glowing neon sign!" So, it paints a fake neon sign on a blank wall. It sees a car and thinks, "At night, cars have bright headlights," so it paints glowing headlights on a bush.

In the world of AI, this is called hallucination. The AI is inventing objects (like fake traffic lights or car headlights) where they don't actually exist. This is a disaster for self-driving cars, because if the car's brain thinks a bush is a traffic light, it might stop in the middle of the road for no reason.

This paper presents a new "Chef's Assistant" (the AI framework) that stops the robot from making these mistakes. Here is how it works, broken down into simple concepts:

1. The Problem: The "Over-Enthusiastic Artist"

Existing AI models are like artists who are too eager to please. When asked to turn a day photo into a night photo, they look at the "Night Style" (dark, glowing lights) and try to copy it everywhere. They don't understand that only cars have headlights and only intersections have traffic lights. They paint lights on trees and signs on empty roads, confusing the self-driving car's brain.

2. The Solution: A Two-Part Security System

The authors built a system with two main tools to catch and stop these fake objects.

Tool A: The "Spotter" (The Dual-Head Discriminator)

Imagine a security guard at a museum.

• Old Guard: Only checks if the painting looks "real" (is the lighting right? is the color dark?). If the fake neon sign looks cool, the guard says, "Good job!"
• New Guard (The Spotter): This guard has a second job. They also have a map of what should be in the room. If the artist paints a neon sign on a blank wall, the Spotter points and says, "Hey! There is no sign in the original photo! That is a fake!"
• How it works: The AI uses a "pseudo-map" (a guess at where objects are) to check the new night photo. If it sees a "traffic light" feature in a place where there was no traffic light before, it flags it as a hallucination.

Tool B: The "Anchor" (Target-Class Prototypes)

Now, how do we stop the artist from painting those fake lights?

• Imagine you have a "Gold Standard" photo of a real car's headlight and a real traffic light. These are your Anchors.
• When the AI tries to paint a fake headlight on a bush, the system compares that bush-painting to the Gold Standard headlight.
• The system says, "Wait! This feature on the bush is trying to look like a headlight, but it's not attached to a car. It's too close to the 'Headlight' anchor."
• The AI is then punished (mathematically pushed away) for making the bush look like a headlight. It learns: "Headlights only belong on cars. If I see a headlight feature on a bush, I must erase it."

3. The Process: Walking Step-by-Step

Instead of jumping from "Day" to "Night" in one giant leap (which causes the AI to get confused and hallucinate), this system walks there slowly.

• It takes the day photo.
• It makes a small change to make it slightly darker.
• It checks: "Did I accidentally paint a fake light?"
• If yes, it fixes it immediately.
• It repeats this many times until it reaches a perfect, realistic night scene.

Why Does This Matter?

The results are impressive. When they tested this on the BDD100K dataset (a huge collection of driving videos):

• Before: Self-driving car detectors trained on these fake night photos were confused and missed real objects.
• After: The detectors became 15.5% more accurate.
• The Big Win: For tricky things like traffic lights, the accuracy jumped by 31.7%.

The Bottom Line

Think of this paper as teaching an AI to be a disciplined translator rather than a creative writer.

• Creative Writer: "It's night! Let's add some magic lights everywhere!" (Result: Confusion).
• Disciplined Translator: "It's night. The car has lights. The sign is dark. The tree is just a tree. No magic lights on the tree." (Result: Safety).

By catching the AI when it tries to "hallucinate" new objects, this method ensures that the translated night scenes are safe, accurate, and ready to help self-driving cars see clearly in the dark.

Technical Summary

1. Problem Statement

Unpaired image-to-image (I2I) translation is crucial for adapting annotated datasets from daytime to nighttime conditions to support downstream tasks like object detection and semantic segmentation in autonomous driving. However, existing methods (GAN-based and Diffusion-based) suffer from a critical flaw: Target-Class Hallucination.

• The Issue: In unpaired translation, models often incorrectly synthesize objects belonging to target classes (e.g., traffic signs, vehicles, traffic lights) in background regions where they do not exist.
• The Cause: Conventional discriminators rely on global style cues (e.g., darkness, light sources) to distinguish real from fake. This inadvertently encourages generators to insert artifacts like fake taillights or traffic signals to mimic the "nighttime style," leading to semantic inconsistencies.
• The Consequence: These hallucinations introduce label noise and false positives, significantly degrading the performance of downstream models trained on the translated data. Existing instance-aware methods focus on preserving annotated foreground objects but fail to constrain unannotated background regions, allowing hallucinations to proliferate.

2. Methodology

The authors propose a novel framework built upon the Schrödinger Bridge (UNSB) multi-step translation model. The core innovation lies in explicitly detecting and suppressing target-class hallucinations through two integrated components:

A. Schrödinger Bridge Multi-Step Translation

Instead of a single-step mapping, the generator models translation as a sequence of stochastic transport steps (Markov chain). It progressively refines the image from the source domain ($x_0$) to the target domain ($x_N$) through intermediate states. This formulation allows for smoother domain shifts and greater diversity in translation.

B. Target-Class Hallucination Segmentation (Detection)

To detect hallucinations, the authors introduce a Dual-Head Discriminator:

1. Style Head ($D_{sty}$): Evaluates global realism (standard adversarial loss).
2. Segmentation Head ($D_{seg}$): Explicitly predicts semantic segmentation maps to identify hallucinated content in background regions.
   • Training Strategy: Since datasets typically only provide bounding boxes (not pixel-level masks), the authors use a foundation model (SAM2) to generate pseudo segmentation masks using bounding boxes as prompts.
   • Hallucination Loss ($L_{hl}$): This loss penalizes the prediction of target classes in unannotated (background) regions. It forces the model to suppress false activations of classes like "traffic light" or "car" in areas where no ground-truth annotation exists.

C. Intermediate Hallucination Suppression (Correction)

To actively remove hallucinated features during the translation trajectory, the authors employ Prototype-Based Contrastive Learning:

• Target-Class Prototypes: The system aggregates features of annotated target-domain objects to create stable "semantic anchors" (prototypes) for each class.
• Contrastive Loss ($L_{supp}$): During intermediate translation steps, if a pixel in the background is detected as resembling a target class (a hallucination), its feature vector is pushed away from the corresponding class prototype.
  • Mechanism: An InfoNCE loss is applied where the hallucinated feature is the anchor, the corresponding source feature is the positive, and the class prototypes serve as additional negative samples. This enforces a clear semantic boundary between background and foreground.

3. Key Contributions

1. Hallucination-Suppression Framework: A multi-step I2I framework that explicitly detects and suppresses target-class hallucinations, ensuring semantic consistency between translations and original annotations.
2. Hallucination-Aware Discriminator: A dual-head discriminator that integrates a segmentation head trained on pseudo-labels (derived from bounding boxes) to penalize semantic inconsistencies rather than just style mismatches.
3. Prototype-Based Suppression: A novel mechanism using class-specific prototypes as semantic anchors to repel hallucinated features from target classes via contrastive learning.
4. Empirical Validation: Demonstrated state-of-the-art performance in reducing hallucinations and improving downstream detection accuracy across diverse datasets.

4. Experimental Results

The method was evaluated on the BDD100K (Day-to-Night) and KITTI $\to$ Cityscapes (Cross-Dataset) benchmarks.

• Downstream Object Detection (BDD100K):

  • The proposed method improved the mean Average Precision (mAP) by 15.5% compared to the baseline domain adaptation (Lower Bound).
  • It achieved a 31.7% gain for traffic lights, a class highly prone to hallucination.
  • Notably, the method outperformed the "Upper Bound" (a detector trained on real nighttime data) for several classes (Truck, Bike, Rider, Person), a feat not achieved by prior work.
  • It surpassed the previous state-of-the-art (InstaFormer) by 13.1% in overall mAP.

• Cross-Dataset Adaptation (KITTI $\to$ Cityscapes):

  • Achieved the highest accuracy across most object classes, demonstrating robustness in cross-domain scenarios.

• Qualitative Results:

  • Visual comparisons show the method effectively eliminates fake taillights, spurious traffic signals, and ghost vehicles that plague other methods (e.g., CycleGAN, CUT, UNSB), while maintaining realistic nighttime lighting.

• Ablation Study:

  • Removing both the hallucination segmentation loss ($L_{hl}$) and suppression loss ($L_{supp}$) resulted in performance barely above the lower bound, proving that hallucination control is critical.
  • The full model achieved the best results, confirming the synergy between detecting hallucinations and pushing features away from prototypes.

5. Significance

This paper addresses a fundamental limitation in unpaired image translation: the trade-off between style transfer and semantic integrity. By shifting the focus from merely "making the image look like night" to "ensuring the night image contains only the objects that were there during the day," the authors provide a robust solution for domain adaptation in safety-critical applications like autonomous driving. The ability to generate high-fidelity synthetic data without introducing label noise significantly enhances the reliability of downstream perception models, potentially reducing the need for expensive nighttime data collection and annotation.