How Well Does GPT-4o Understand Vision? Evaluating Multimodal Foundation Models on Standard Computer Vision Tasks

1. Problem Statement

Multimodal Foundation Models (MFMs) like GPT-4o, Gemini, and Claude have demonstrated remarkable progress in language and visual question answering (VQA). However, their ability to perform dense, pixel-level visual understanding (e.g., segmentation, depth estimation, surface normals) remains unclear.

Existing benchmarks for MFMs suffer from three main limitations:

1. Text-Centric Bias: Most benchmarks (e.g., VQA) rely on text outputs, making it difficult to isolate visual capabilities from language capabilities.
2. Incompatibility with Specialists: Standard vision tasks require dense outputs (masks, coordinates, maps) that MFMs cannot natively generate, preventing direct comparison with state-of-the-art (SOTA) vision-only models.
3. Lack of Geometric Evaluation: Current evaluations focus heavily on semantics (classification) while neglecting 3D geometry (depth, normals) and grouping.

The paper aims to establish a standardized, quantitative benchmark to measure MFMs on classical computer vision tasks, comparing them directly against specialized vision models.

2. Methodology: Prompt Chaining Framework

Since MFMs are primarily trained to output text and cannot natively generate segmentation masks or 3D coordinates, the authors developed a Prompt Chaining framework. This method translates standard vision tasks into a series of text-promptable sub-tasks, allowing MFMs to solve them via API calls.

The core strategy involves decomposing complex tasks into simpler classification or ranking sub-tasks:

• Object Detection: Instead of asking for coordinates directly (which fails), the model performs a recursive grid search. It identifies the presence of an object in grid cells, discards empty cells, and recursively zooms in on relevant regions to refine bounding boxes.
• Semantic Segmentation: The image is divided into superpixels (using SLIC clustering). The MFM classifies the semantic label of each superpixel. To handle context, the model is fed a "multi-scale pyramid" (the crop, surrounding context, and full image) for each superpixel.
• Grouping (Instance Segmentation): Given a query point, the model performs a Breadth-First Search (BFS) on a superpixel graph. It iteratively decides whether adjacent superpixels belong to the same object as the query point.
• Depth & Surface Normal Prediction: Since absolute prediction is hard, the model performs pairwise ranking. It compares pairs of superpixels to determine relative depth or alignment with basis vectors (up, down, left, right). These pairwise rankings are then globalized using an optimization objective (minimizing a loss function) to reconstruct a dense map.

Control Baselines: To ensure fair comparison, the authors introduced:

• Vision Specialist + Chain: Applying the same prompt-chain constraints (superpixels, grid search) to SOTA vision models (e.g., DETR, OneFormer) to isolate the impact of the prompting method vs. the model's inherent capability.
• Oracle + Chain: Assuming ground-truth answers for sub-tasks to determine the theoretical upper bound of the pipeline.
• Blind Guess: Prompting with blank images to measure bias.

3. Key Contributions

1. First Standardized Benchmark for MFMs on Dense Vision Tasks: The paper evaluates MFMs on six standard tasks: Image Classification, Object Detection, Semantic Segmentation, Grouping, Depth Prediction, and Surface Normal Prediction.
2. Prompt Chaining Framework: A novel, API-compatible methodology that enables MFMs to solve tasks requiring dense pixel-wise outputs, bridging the gap between text-based LLMs and vision tasks.
3. Comprehensive Evaluation: Evaluation of 7 leading MFMs (GPT-4o, o4-mini, Gemini 1.5/2.0, Claude 3.5, Qwen2-VL, Llama 3.2) and reasoning models (o1, o3) against SOTA vision specialists.
4. Analysis of Failure Modes: Identification of systematic errors, such as "blurry vision" (poor fine-grained localization), coordinate frame ambiguities, and hallucinations in image generation.

4. Key Results

A. General Performance

• MFMs are "Respectable Generalists" but not Specialists: MFMs perform significantly better than random guessing and show strong semantic understanding, but they lag behind SOTA specialist models in all tasks, even when specialists are constrained by the same prompt-chain limitations.
• Semantic vs. Geometric: MFMs perform notably better on semantic tasks (classification, segmentation) than geometric tasks (depth, surface normals).
• Top Performer: GPT-4o achieves the best performance among non-reasoning models, securing the top spot in 4 out of 6 tasks.

B. Task-Specific Findings

• Classification: MFMs show robustness to corruptions and domain shifts, with GPT-4o and Gemini 2.0 Flash leading.
• Object Detection: GPT-4o achieves an AP@0.5 of ~60.6, significantly outperforming other MFMs but still trailing Co-DETR (SOTA) which scores ~91.
• Segmentation: GPT-4o achieves ~44.9 mIoU, while OneFormer (SOTA) achieves ~65.5.
• Geometric Tasks (Depth/Normals):
  • MFMs struggle significantly here. Many models show negative correlations in surface normal predictions (e.g., confusing left/right axes).
  • Reasoning Models (o1, o3, o4-mini): These models show marked improvements in geometric tasks. o4-mini and o3 significantly outperform GPT-4o in depth and surface normal estimation, suggesting that reasoning capabilities enhance 3D spatial understanding.

C. Prompt Sensitivity & Failure Modes

• Prompt Sensitivity: Better models (like GPT-4o) are less sensitive to prompt variations than weaker models.
• Failure Modes:
  • Blurry Vision: MFMs struggle with fine-grained details; performance drops significantly without multi-scale context.
  • Ceiling Bias: Models consistently infer that objects higher in the image are deeper (a common bias in indoor scenes).
  • Image Generation: When GPT-4o attempts to generate dense outputs (masks/depth maps) directly, it suffers from spatial misalignment and hallucinations, making it less reliable than the prompt-chaining text approach.

5. Significance and Conclusion

This paper provides the first rigorous, apples-to-apples comparison of Multimodal Foundation Models against specialized vision models on standard computer vision tasks.

• Current State: While MFMs are powerful generalists with impressive semantic capabilities, they are not yet ready to replace specialist models for tasks requiring precise geometric understanding or dense pixel-level prediction.
• Future Direction: The emergence of reasoning models (o1, o3, o4-mini) offers a promising path forward, as they demonstrate the ability to improve geometric reasoning through chain-of-thought processes.
• Benchmarking Utility: The proposed prompt-chaining framework serves as a vital tool for the community to evaluate future MFMs, ensuring that claims of "visual understanding" are backed by performance on standard, dense vision tasks rather than just text-based VQA.

The authors conclude that while MFMs are closing the gap in semantic understanding, a significant gap remains in geometric reasoning, which may require architectural changes or specific training on dense vision tasks rather than just scaling up multimodal pre-training.