Technical Summary: Evolution of Accuracy and Visual-Cognitive Errors in a Decade of Vision-Language AI Models

1. Problem Statement

While Vision-Language Models (VLMs) have demonstrated remarkable progress in visual reasoning over the last decade, existing evaluations rely heavily on simple scene datasets (e.g., MS-COCO) that lack complex human interactions and behaviors. Consequently, current benchmarks may not adequately capture the challenges of everyday visual inference involving social dynamics. Furthermore, prior research has largely focused on aggregate performance metrics (e.g., BLEU, CIDER) rather than dissecting the specific types of visual-cognitive errors that models commit or how these error profiles have evolved. There is a need to understand whether models have truly achieved human-level reasoning capabilities in complex scenarios and to identify which specific error types persist despite architectural advancements.

2. Methodology

2.1 Datasets

The study utilizes two distinct datasets:

• Complex Social Behavior (CSB) Dataset: A novel dataset comprising 100 images depicting intricate social interactions and complex human behaviors, primarily sourced from movie scenes.
• MS-COCO Dataset: A control set of 100 images randomly selected from the standard MS-COCO dataset, featuring simpler scenes (e.g., objects, simple activities) without complex social dynamics.

2.2 Models Evaluated

The authors evaluated nine vision-language models spanning the period from 2017 to 2025:

• Pre-MLLMs (4 models): Up-Down Captioner (UDC), Meshed Memory Transformer (M2M), OSCAR, and OFA. These typically utilize ResNet-101 backbones with LSTM or Transformer encoders/decoders.
• Multi-Modal Large Language Models (MLLMs) (5 models): GPT-4, Gemini, GPT-4o1, GPT-5.1, and Gemini3. These utilize Transformer architectures trained on vast image-text pairs with reinforcement learning from human feedback.

2.3 Human Benchmarking

• Gold Standard Generation: For each image, 20 human descriptions were collected via Amazon Mechanical Turk. A separate group of 50 workers ranked these descriptions; the top-ranked description served as the "gold standard."
• Human Performance Baseline: The top four and bottom four human descriptions were retained to establish performance bounds for comparison against models.

2.4 Evaluation Metrics

• Semantic Similarity: The study employs Gemini-Similarity Index (Gemini-SI) and GPT-Similarity Index (GPT-SI), which calculate the cosine similarity of text embeddings generated by large language models. Validation showed these metrics correlate more strongly with human ratings than BERTScore.
• Error Taxonomy: Five specific visual-cognitive error types were defined and analyzed:
  1. Object Detection: Failure to detect relevant objects present in the scene.
  2. Object Recognition: Misclassification of detected objects.
  3. Hallucination: Describing objects not present in the image or misclassifying irrelevant objects as relevant.
  4. Scene Understanding: Correctly recognizing an object but omitting it from the description.
  5. Spatial Dependence: A novel metric measuring the degree to which models and humans rely on the same image regions for description generation. This was assessed using an occlusion masking paradigm (masking 1/3 of the image) to determine if the loss of specific regions degraded the description similarly for humans and models.

2.5 Statistical Analysis

Due to the limited sample size (200 images), the authors utilized bootstrapping (10,000 resamples) to estimate variability and generalization. Significance levels were corrected for the False Discovery Rate (FDR) using the Benjamini-Hochberg method.

3. Key Results

3.1 Performance Evolution

• Complex vs. Simple Scenes: Pre-MLLMs performed significantly worse on the CSB dataset compared to MS-COCO, often falling below the performance of the bottom-ranked human descriptions. In contrast, MLLMs achieved accuracy on the CSB dataset comparable to the top-ranked human descriptions, effectively eliminating the performance gap between simple (MS-COCO) and complex (CSB) scenes.
• Training Data Bias Check: To rule out the possibility that MLLMs simply memorized the movie scenes used in CSB, the authors tested models on an in-house dataset of unseen images. MLLMs maintained high performance similar to humans, while pre-MLLMs (specifically OFA) remained significantly lower, confirming that the improvement is due to model capability rather than data memorization.

3.2 Error Analysis

• Error Reduction: MLLMs have almost eliminated detection, recognition, and hallucination errors across both datasets. Pre-MLLMs exhibited significantly higher rates of these errors on the CSB dataset compared to MS-COCO.
• Persistent Spatial Dependence: The only error type that remains significant for MLLMs is Spatial Dependence Error (SDE). While MLLMs have reduced SDE compared to earlier models, they still occasionally rely on different image regions than humans to generate descriptions.
• Impact on Accuracy: Detection, recognition, and hallucination errors had the highest negative impact on semantic similarity scores (lowering Gemini-SI to 0.38–0.41). Scene understanding errors had a moderate impact, while spatial dependence errors had the least impact on overall similarity scores (0.56).

4. Contributions

• CSB Dataset: Introduction of a benchmark specifically designed to evaluate visual reasoning in complex social interactions, addressing the limitations of standard datasets like MS-COCO.
• Error Taxonomy: A comprehensive classification of visual-cognitive errors (detection, recognition, hallucination, scene understanding, and spatial dependence) applied across a decade of model evolution.
• Methodological Innovation: Development of a masking paradigm to quantify spatial dependence errors, allowing for the comparison of where models and humans focus their attention without requiring access to internal model weights (model-agnostic).
• Longitudinal Analysis: A unique evaluation of nine models spanning 2017–2025, documenting the transition from task-specific pre-MLLMs to general-purpose MLLMs.

5. Significance and Claims

The paper claims that while MLLMs have successfully bridged the gap in scene description accuracy between simple and complex social scenes—reaching human-level performance on complex tasks—they have not fully aligned with human visual reasoning processes. Specifically, the persistence of spatial dependence errors suggests that MLLMs may still rely on different visual cues or regions than humans when describing complex interactions.

The authors conclude that their findings provide a more thorough evaluation of VLM advancements, highlighting that while "factual" errors (detection, recognition, hallucination) have been largely overcome, deeper cognitive misalignments regarding where information is extracted from an image remain. The study emphasizes that evaluating models solely on simple datasets (MS-COCO) masks the specific challenges and error profiles associated with complex human behavior, and that future improvements must address these spatial and reasoning misalignments.