🤖 AI

CityGen: Structure-Guided City-Style Synthesis for Cross-City Autonomous Driving

This paper introduces CityTransfer-Bench, a geographically disjoint benchmark for evaluating cross-city generalization, and proposes CityGen, a diffusion-based generative framework that enables zero-label city adaptation through HD-map-conditioned synthesis to improve autonomous driving robustness across diverse urban environments.

Original authors: Zezhong Qian, Zhao Yang, Lu Tan, Zhihao Yan, Weiyi Hong, Haizhuang Liu, Yawei Jueluo

Published 2026-05-29

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Zezhong Qian, Zhao Yang, Lu Tan, Zhihao Yan, Weiyi Hong, Haizhuang Liu, Yawei Jueluo

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to drive a car. You train it in Singapore, a city with specific buildings, road signs, and weather. The robot learns perfectly there. But then, you take that same robot to Boston. Suddenly, the roads look different, the buildings have a different style, and the traffic moves differently. The robot gets confused and starts making mistakes. This is the "cross-city" problem: a self-driving car trained in one city often fails in another.

This paper introduces two main things to fix this: a new test to measure the problem, and a new tool to solve it.

1. The New Test: "CityTransfer-Bench"

Think of this as a strict final exam. Previously, researchers would mix data from different cities together, which made it hard to tell if a robot was truly learning to adapt or just memorizing the mix.

The authors created a "geographically disjoint" test. They trained the robots only on data from Singapore and then tested them only on data from Boston. This ensures the robot has never "seen" Boston before. It's like teaching a student only in English and then testing them in French to see if they can truly generalize their language skills. This test checks three skills:

Perception: Can it see cars and people?
Segmentation: Can it understand where the road ends and the sidewalk begins?
Planning: Can it decide how to steer and brake safely?

2. The New Tool: "CityGen" (The Digital Translator)

To help the robot pass this exam, the authors built CityGen. Imagine CityGen as a magical digital translator that can change the "skin" of a scene without changing its "skeleton."

Here is how it works, using a simple analogy:

The Skeleton (HD-Maps): Every city has a rigid structure: where the lanes are, where the stop signs are, and where the cars are located. This is the "skeleton." CityGen uses high-definition maps to lock this skeleton in place. It ensures that if a car is in the left lane in the original photo, it stays in the left lane in the new photo. This keeps the geometry perfect.
The Skin (City Style): Cities have a "vibe" or "skin." Singapore might have lush green trees and tropical light, while Boston has brick buildings and autumn leaves. CityGen uses a special AI (a diffusion model) to strip away the Singapore "skin" and paint on a Boston "skin."
The Magic Trick (Zero-Label): Usually, to teach a robot about Boston, you need humans to manually label thousands of Boston photos (drawing boxes around cars, etc.). This is expensive and slow. CityGen is special because it is "zero-label." It looks at unlabeled photos of Boston (just raw images) to learn the "style" (the colors, the textures, the lighting). It then applies that style to the Singapore skeleton.

The Result:
CityGen takes a photo from Singapore and turns it into a photo that looks exactly like it was taken in Boston, but with the exact same road layout and car positions.

Why This Matters

The paper shows that when they use these "fake" Boston photos to train the robot, the robot becomes much better at driving in the real Boston.

Before CityGen: The robot trained on Singapore data performed poorly in Boston (like a student failing a French test).
After CityGen: The robot trained on Singapore data plus CityGen's synthetic Boston data performed much better. It learned to recognize that a "car" is still a "car," even if the car is painted a different color or the streetlights look different.

The Bottom Line

The authors created a new way to test if self-driving cars can handle new cities, and they built a tool that acts like a "style filter." This filter takes familiar driving scenes and re-dresses them in the clothes of a new city, teaching the AI to be flexible without needing expensive human labels for every new location. It's a way to make self-driving cars more "world-ready" by practicing in a virtual, style-translated version of the world.

Problem Statement
Autonomous driving systems are typically trained and evaluated within limited geographic regions, leading to significant performance degradation when deployed in unseen cities. This "cross-city" failure stems from domain shifts in road layouts, visual appearance, traffic patterns, and environmental conditions. Existing solutions, such as domain adaptation, data augmentation, and synthetic data generation, often rely on labeled target data, city-specific annotations, or task-specific designs. These dependencies limit scalability and effectiveness for holistic evaluation across perception, segmentation, and planning tasks. Furthermore, current benchmarks often lack strict geographic separation, failing to rigorously assess true city-level generalization.

Methodology: CityGen and CityTransfer-Bench
To address these challenges, the authors introduce two primary components:

CityTransfer-Bench: A geographically disjoint benchmark designed to evaluate cross-city generalization. Built upon the nuScenes dataset, it strictly separates training and testing data by location: models are trained exclusively on data from three Singapore regions (One-North, Queenstown, and Holland Village) and evaluated on the Boston Seaport area. The benchmark assesses three core autonomous driving tasks: perception (detection), segmentation, and planning.
CityGen: A diffusion-based generative framework for zero-label city adaptation. It synthesizes realistic urban scenes that mimic the target city's appearance while preserving the source city's geometric structure. The method operates through a dual-conditioning mechanism:
- Structure-Guided Geometric Control: The framework utilizes High-Definition (HD) maps to enforce geometric consistency. It projects city-invariant structural primitives (lane boundaries, centerlines, and 3D bounding boxes of agents) onto multi-view camera planes to create a structural control tensor ( $M_{hdmap}$ ). This tensor is injected into the diffusion model via a structural control branch, ensuring that lane topology and object layouts remain stable across domains.
- City-Level Visual Prompts: To capture target-city appearance (e.g., architecture, vegetation, lighting), the system constructs an "unlabeled city-style library" from target-domain videos. A Vision-Language Model (VLM), specifically InternVL, extracts scene captions from these videos to form a set of style descriptors. These descriptors act as text-conditioning signals during the diffusion process.
- Synthesis Process: The model employs a Diffusion Transformer (DiT) architecture. It denoises latent variables conditioned on both the structural map ( $M_{hdmap}$ ) and a sampled style descriptor ( $y$ ). This allows for the generation of multi-view urban scenes that maintain the source geometry (preserving existing annotations) while adopting the target city's visual style.

Key Contributions

CityTransfer-Bench: The first benchmark specifically designed to evaluate city-level generalization across perception, segmentation, and planning tasks using a geographically disjoint split.
CityGen Framework: A novel diffusion-based approach for zero-label city adaptation that decouples structural layout from visual style. It enables semantic-preserving style transfer without requiring additional annotations for the target city.
Empirical Validation: Large-scale experiments demonstrating that CityGen-generated data significantly improves downstream robustness on unseen cities, establishing a scalable foundation for cross-domain autonomous driving.

Experimental Results
Evaluated on the CityTransfer-Bench using StreamPetr (detection), Cross-View Transformer (segmentation), and UniAD (planning), CityGen consistently outperformed existing enhancement methods and baselines trained only on source data:

Detection: Improved mean Average Precision (mAP) from 10.18 to 13.06 and Normalized Detection Score (NDS) from 28.05 to 33.95 compared to the source-only baseline.
Segmentation: Achieved substantial gains, increasing IoU40 from 43.47 to 50.84 and IoU50 from 41.97 to 49.39, significantly narrowing the gap to the upper-bound baseline trained on both cities.
Planning: Reduced the average collision rate to 0.350% and the average L2 error to 0.984, outperforming the source-only baseline.
Ablation Studies: Results confirmed that both target-city style alignment and VLM-based semantic style encoding are essential; using source-city captions or removing VLM encoding led to performance degradation. Additionally, CityGen maintained strong robustness even when the labeled source training data was reduced to 20%, demonstrating improved label efficiency.

Significance and Claims
The paper claims that CityGen establishes a scalable and label-efficient foundation for generalizable autonomous driving. By synthesizing data that is structurally consistent yet visually diverse, the method effectively exposes downstream models to a broad spectrum of target-city visual conditions without the prohibitive cost of manual annotation. The authors posit that this structure-guided synthesis approach overcomes the limitations of previous methods that struggle with large-scale, real-world cross-city generalization due to inherent differences in traffic patterns and road structures. The work provides a unified protocol for assessing city-level transferability, addressing a critical gap in the scalability and reliability of autonomous driving systems.

1. The New Test: "CityTransfer-Bench"

2. The New Tool: "CityGen" (The Digital Translator)

Why This Matters

The Bottom Line

More like this