💻 computer science

StereoGenBench: A Synthetic Multi-Camera Benchmark for Stereo Generation under Controlled Baseline Regimes

The paper introduces StereoGenBench, a synthetic Unreal Engine benchmark featuring a rigid six-camera array that provides scene-paired, calibrated multi-baseline stereo data with complete geometric metadata (RGB, metric depth, intrinsics, and poses) to enable controlled evaluation of stereo generation under varying baseline regimes.

Original authors: Yangzhi Cui, Feng Qiao, Nathan Jacobs

Published 2026-05-25

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Yangzhi Cui, Feng Qiao, Nathan Jacobs

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot how to see the world in 3D, just like humans do with two eyes. To do this, the robot needs to understand stereo vision: how the distance between its eyes (the "baseline") changes how objects look.

The problem is that most existing training data for these robots is like a photo album where every picture was taken with the exact same camera setup. The distance between the lenses never changes, the zoom never changes, and the depth is often guessed rather than measured. If you train a robot only on these fixed photos, it gets confused when it sees a real-world scene where the camera spacing is different. It's like teaching someone to drive only on a straight, empty highway at 30 mph, and then expecting them to handle a winding mountain road at 100 mph.

StereoGenBench is a new, synthetic (computer-generated) "driving school" designed to fix this. Here is how it works, using simple analogies:

1. The "Six-Eye" Camera Rig

Instead of using a standard two-camera setup, the researchers built a virtual rig with six cameras lined up side-by-side, like a row of security cameras or a row of eyes on a bug.

The Magic: Because there are six cameras, they can create 15 different pairs of "left eye" and "right eye" views from the exact same scene.
The Control: They can mix and match these cameras to simulate tiny distances between eyes (like human eyes) or huge distances (like cameras on opposite sides of a room). This allows them to test how well a robot handles different "eye spacings."

2. The "Perfectly Labeled" World

In the real world, it's very hard to know exactly how far away an object is or exactly how the camera moved.

The Solution: StereoGenBench is built inside a game engine (Unreal Engine). Because the computer created the scene, it knows the exact truth.
The Data: For every single frame of video, the dataset provides:
- The RGB color image (what the eye sees).
- Metric Depth: The exact distance to every pixel (like a laser scan).
- Intrinsics: The exact "lens" settings (focal length).
- Poses: The exact position and angle of every camera.
The Analogy: Imagine a magic mirror that not only shows you a reflection but also tells you the exact distance to every object in the reflection, the exact type of glass used, and the exact angle the mirror is tilted. That is what this dataset provides.

3. Three Different "Exams"

The paper doesn't just dump the data; it sets up three specific ways to test the robots (AI models), depending on what information they are allowed to use:

The "Cheat Sheet" Exam (Tier G0): The robot is given the left image plus the exact answer key (the true depth or a warped version of the right image). This tests if the robot can render a picture correctly when it already knows the geometry.
The "Hint" Exam (Tier G1): The robot gets the left image and a hint about the camera settings (e.g., "the eyes are 10cm apart"), but it doesn't know the exact depth. This tests if the robot can use camera metadata to guess the 3D shape.
The "Blind" Exam (Tier G2): The robot gets only the left image and has to guess the right image using its own internal brainpower. This is the hardest test, checking if the robot has learned the general rules of 3D vision or if it just memorized the training data.

4. Why This Matters

The paper shows that when you test these robots on this new benchmark, their performance changes drastically based on the "eye spacing" (baseline).

The Discovery: Some robots that look great on standard tests fall apart when the camera spacing gets wider. They lose their sense of scale.
The Analogy: It's like a musician who can play a song perfectly in a small room but gets lost when the room gets huge. StereoGenBench proves that the "size of the room" (the baseline) is a critical variable that current AI models often ignore.

5. What They Released

The authors didn't just write a paper; they opened the doors to the whole "driving school." They released:

The dataset (over 8,000 scenes).
The code to generate new scenes.
The code to run the tests.
A list of "reference scores" showing how current top AI models perform on this new test.

In short: StereoGenBench is a controlled, synthetic playground where researchers can finally isolate and measure how well AI understands 3D space when the camera settings change, something that was impossible to do cleanly with real-world data before.

Technical Summary: StereoGenBench

Problem Statement

Stereo image and video generation, stereo geometry estimation, and condition-controlled view synthesis fundamentally require paired data where the variables governing binocular geometry—specifically camera baseline, intrinsics, scene depth, and camera motion—are known and controllable. Existing resources fall short of this requirement in two primary ways:

Real-world benchmarks (e.g., KITTI, Cityscapes, DrivingStereo) fix the camera rig baseline and intrinsics at capture time. While valuable, they conflate a method's stereo-scale behavior with the specific scene distribution and fixed hardware configuration used during capture. Metric depth is often sparse or absent.
Synthetic resources (e.g., Scene Flow, SimStereo, StereoCarla) provide dense rendered supervision but typically retain fixed rig configurations and intrinsics. They do not offer scene-paired, calibrated multi-baseline ground truth where the baseline is a controlled variable rather than a fixed nuisance parameter.

The authors argue that the baseline is not a nuisance variable; for a rectified stereo pair, disparity scales as $d = B f_x z^{-1}$ . Training and evaluating methods only at a fixed baseline obscures whether performance changes with realized baseline or whether a generated view matches a specified target camera. There is a lack of resources that provide scene-paired, calibrated multi-baseline right-view ground truth with jointly recorded intrinsics, dense metric depth, and per-frame poses in a single controlled source.

Methodology: StereoGenBench

StereoGenBench is a synthetic benchmark generated using Unreal Engine, designed to make baseline-regime sensitivity and target-camera consistency measurable under matched scene content.

Data Generation Pipeline

Six-Camera Rig: Each scene is rendered using a rigid, rectified lateral array of six synchronized cameras. This configuration yields up to $\binom{6}{2} = 15$ calibrated view pairs per scene.
Controlled Variables: Unlike previous synthetic datasets, StereoGenBench varies four axes within a single source while preserving paired ground truth:
- Baseline: Adjacent spacings are sampled from specific regimes.
- Intrinsics: Focal length and sensor dimensions are sampled independently and recorded.
- Depth: Dense metric depth is rendered for every camera.
- Pose: Per-frame six-camera poses are recorded.
Baseline Sampling Families:
- IPD_Gaussian: Samples adjacent spacings from a truncated Gaussian centered at 6.38 cm (inter-pupillary distance), reflecting narrow-baseline regimes.
- Uniform: Samples adjacent spacings from a uniform distribution $U[1.0, 150.0]$ cm, defining a wide-baseline evaluation branch.
- Pairwise_Uniform: A train-only family designed to improve all-pairs baseline coverage when the six-camera rig is expanded to all 15 pairs.
Scene Construction: Scenes are constructed via a proposal-and-filter process involving 25 maps (indoor, outdoor, urban, natural, sci-fi, stylized), 15 character FBX files, and 30 action FBX files. Trajectories are generated by sampling valid viewpoints and connecting them into 81-frame paths, with restart-and-splice procedures for invalid transitions.
Output: Each scene includes six RGB videos, six metric-depth videos, and metadata files (baseline.json, trajectory.json) containing intrinsics, baselines, and poses. The resolution is 1280 $\times$ 1280 at 15 fps.

Benchmark Protocol

The benchmark evaluates right-view generation under three distinct inference conditions (tiers) to separate different research questions:

Tier G0 (Oracle Geometry-Conditioned): Methods receive the left view ( $I_L$ ) plus target-view geometry (e.g., ground-truth disparity, depth, or a warped right view). This measures synthesis quality under known geometry.
Tier G1 (Calibrated Target-Camera): Methods receive $I_L$ and target-camera metadata (baseline $B$ and intrinsics $K$ ) but no ground-truth depth.
Tier G2 (Unaligned Monocular): Methods receive only $I_L$ and rely on their own implicit geometry.

Evaluation Metrics

Results are reported using three groups of metrics:

Reconstruction Quality: PSNR, SSIM, and LPIPS comparing generated ( $\hat{I}_R$ ) and rendered target ( $I_R$ ) views.
Stereo-Geometry Diagnostics:
- $E_{Match}$ : Measures the change in the set of stereo-matchable left-image structures when replacing the rendered right view with the generated one.
- P-PSNR: A target-free consistency diagnostic measuring local texture similarity.
- SD (Scale Deviation): Measures disparity-scale fidelity by fitting $D_{gt} \approx a \hat{D} + b$ and reporting $|a - 1|$ .
Distributional Metrics: FID (image) and FVD (video) comparing generated and rendered target distributions.

Key Contributions

The paper contributes:

StereoGenBench Dataset: A scene-paired synthetic resource featuring a six-camera calibrated rig, dense metric depth, intrinsics, per-pair baselines, and per-frame poses.
Split Design: A separation of narrow IPD-scale evaluation, wide-baseline evaluation, and a train-only all-pairs family to improve baseline coverage.
Benchmark Protocol: A tiered evaluation framework distinguishing between oracle geometry-conditioned, calibrated target-camera, and unaligned monocular inference settings.
Open Release: Public availability of the dataset, evaluation code, reference results, Croissant metadata, and the full generation pipeline, enabling extensions with custom assets.

Reference Results

The authors report results on 739 evaluation scenes (split into seen-map and unseen-map subsets) using representative public methods.

Tier Differences: As expected, oracle geometry-conditioned methods (G0) achieve stronger pixel alignment (higher PSNR/SSIM) because their inputs are anchored to ground truth. Unaligned monocular methods (G2) exhibit larger disparity-scale drift (higher SD) because their outputs are not anchored to the benchmark target camera.
Baseline Sensitivity: The benchmark successfully isolates baseline effects. For example, in the Uniform branch, the SD of the G2 method Mono2Stereo rises monotonically from 2.63 at [1, 10) cm to 50.85 at [100, 150] cm. In contrast, the G0 method GenStereo remains stable (0.03–0.09) across the same bins.
Difficulty: The Uniform branch (wide baseline) is generally harder than the IPD_Gaussian branch, reflecting larger realized disparities and more disocclusion.
Generalization: Unseen-map results show that while matchability ( $E_{Match}$ ) remains relatively stable, distributional metrics (FID/FVD) show larger gaps between seen and unseen maps, indicating sensitivity to visual priors.

Significance and Claims

The paper positions StereoGenBench as a complementary resource to existing real-world and synthetic benchmarks, rather than a replacement. Its primary significance lies in isolating the baseline variable for stereo generation evaluation. By rendering matched scene content under different calibrated baseline regimes, the benchmark allows researchers to:

Determine if performance changes with realized baseline.
Verify if a generated right view matches a specified target camera.
Diagnose whether methods use calibrated target-camera interfaces or produce stereo at a preferred implicit scale.

The authors explicitly state that results on this synthetic dataset should not be treated as direct evidence of real-world performance without separate correlation studies. The benchmark is intended for evaluating geometry-aware tasks (stereo generation, matching, depth estimation) under controlled conditions, not for surveillance, biometric identification, or demographic analysis. The release enables future studies on baseline conditioning, multi-pair stereo generation, and sim-to-real correlation.