Technical Summary: MAPS – A Synthetic Dataset for Probing Vision Models in a Controlled 3D Scene Space

Problem Statement

Modern vision models achieve high aggregate accuracy on standard benchmarks like ImageNet, yet these metrics obscure the specific scene properties driving model predictions. Natural images inherently entangle object identity with confounding variables such as lighting, viewpoint, background, and scale. Consequently, models often exploit spurious correlations or "shortcuts" (e.g., relying on texture over shape or background context over object features) to achieve high performance, rather than learning robust, invariant representations.

Existing robustness benchmarks offer partial solutions but suffer from significant limitations:

• Natural-image datasets (e.g., ImageNet-E, ObjectNet) typically manipulate only 2D image attributes or discrete conditions, failing to disentangle underlying 3D scene structure.
• Synthetic 3D datasets (e.g., CLEVR, 3DIdent) often rely on simple geometric shapes, cover few object classes, or lack alignment with the broad semantic diversity of ImageNet.
• Lack of Control: There is a missing scalable framework that allows for the independent, continuous variation of multiple 3D scene factors across a wide range of real-world object categories to systematically quantify model sensitivity.

Methodology

The MAPS Dataset

The authors introduce MAPS (Manifolds of Artificial Parametric Scenes), a scalable, controllable, and ImageNet-aligned 3D benchmark.

• Mesh Curation: The dataset comprises 2,618 manually curated, photorealistic 3D meshes spanning 560 ImageNet classes. Meshes were sourced from Sketchfab, filtered for photorealism, and validated for legal redistribution (CC-BY/CC0).
• Recognizability Validation: To ensure the meshes are recognizable to standard vision models, the authors employed CMA-ES (Covariance Matrix Adaptation Evolution Strategy). This gradient-free optimization algorithm searched the 9-dimensional scene parameter space to find at least one configuration where a reference model (AlexNet, ResNet-50, or ViT-B/16) correctly classified the mesh. All 2,618 meshes were confirmed recognizable under suitable parameters.
• Semantic Diversity: Analysis of the WordNet structure reveals that MAPS covers diverse semantic clusters (e.g., tools, vehicles, animals) but exhibits a bias toward prototypical, man-made objects over fine-grained animal synsets (e.g., specific dog breeds), reflecting the availability of high-quality 3D assets.

Rendering Pipeline

MAPS provides an open, Blender-based rendering pipeline that enables on-demand image generation. Unlike pre-rendered datasets, MAPS exposes the parametric scene space itself, allowing users to design custom sampling protocols. The pipeline controls nine independent scene factors grouped into three families:

1. Camera: Azimuth, Elevation, Distance, Roll.
2. Lighting: Azimuth, Elevation, Power.
3. Background: Hue, Saturation.

Evaluation Protocol

To evaluate model sensitivity, the authors selected 100 representative ImageNet classes from MAPS. For each mesh in these classes, they rendered 5,000 images using Latin Hypercube Sampling (LHS) to ensure broad coverage of the 9D parameter space with fewer samples than uniform grid sampling.

Sensitivity Analysis:
For each model-mesh pair, the authors fitted linear and polynomial regression models to predict the model's decision margin (the difference between the logit of the correct class and the maximum logit of competing classes) based on the scene parameters.

• Linear Regression: Captures first-order effects.
• Polynomial Regression: Captures non-linear effects (quadratic terms and pairwise interactions).
• Metric: The magnitude of regression coefficients serves as a measure of sensitivity to specific scene factors.

Key Results

1. Universal Failure Axes

Across all 20 evaluated models (spanning CNNs like ResNet, VGG, and Transformers like ViT, Swin), camera distance and camera elevation emerged as the dominant drivers of recognition failure.

• Increased camera distance consistently reduced the decision margin, pushing models toward misclassification.
• Extreme camera elevations (very low or very high angles) disproportionately degraded performance compared to frontal views.
• This failure pattern is consistent across architectures, regardless of their overall ImageNet accuracy, suggesting a shared bias in how these models handle spatial depth and viewpoint.

2. Sensitivity Profiles Reveal Architectural Nuance

While aggregate accuracy often groups models loosely (e.g., "CNNs" vs. "Transformers"), the sensitivity profiles revealed by MAPS offer a finer-grained distinction:

• Clustering: Modern CNNs (e.g., ConvNeXt) and Transformer-based models cluster together in their sensitivity profiles, forming a distinct group from older architectures (e.g., AlexNet, VGG).
• Design Choices Matter: The divergence between older and newer models suggests that fine-grained architectural design choices (e.g., patchify stems, large depthwise kernels, LayerNorm in ConvNeXt) are stronger determinants of sensitivity profiles than the broad CNN-versus-Transformer distinction.
• Hidden Differences: Models with similar ImageNet top-1 accuracy (e.g., ResNet-18 and GoogLeNet) displayed markedly different sensitivities to background saturation and light power, differences invisible to standard accuracy metrics.

3. Internal Representations

The study also visualized internal representations, finding that intermediate layers in several architectures recover the toroidal topology of the scene-parameter manifold (specifically when sweeping background hue and camera azimuth), indicating that the geometric structure of the input space is reflected in the model's latent space.

Significance and Claims

The paper positions MAPS as a programmable, scalable reference frame for probing vision models. Its primary significance lies in:

• Disentangling Causality: By providing ground-truth generative parameters, MAPS allows researchers to attribute model behavior directly to specific scene factors rather than uncontrolled correlations in natural images.
• Beyond Accuracy: It demonstrates that summary metrics like accuracy obscure structured differences in model behavior. Two models can have identical accuracy but rely on fundamentally different visual cues (e.g., one relying on texture, another on shape) to reach that accuracy.
• Mechanistic Interpretability: The dataset supports both behavioral analysis (sensitivity profiles) and mechanistic research (probing the geometry of internal representations along controlled trajectories).
• Future Utility: The authors note that the open rendering pipeline supports future work, such as using differentiable rendering for gradient-based failure mode discovery or comparing representational spaces in artificial and biological vision systems.

The authors maintain a modest stance, acknowledging that MAPS is designed for controlled sensitivity analysis rather than measuring absolute robustness in deployment. They also note current limitations, such as the lack of control over contextual objects, occlusions, or texture variations, and the inherent bias toward prototypical categories due to 3D asset availability.