Technical Summary: BRUCE – Benchmarking Robustness Under Corruption Escalation for Scientific Vision-Language Reasoning

1. Problem Statement

Vision-Language Models (VLMs) have demonstrated strong performance on clean-task benchmarks for tasks such as visual question answering, scientific reasoning, and document analysis. However, their robustness and reliability in real-world scenarios remain poorly understood, particularly when inputs are degraded by distortions such as blur, low contrast, rotation, or cropping. In scientific and mathematical reasoning contexts, even minor visual perturbations can distort diagrams, symbolic structures, or molecular representations, leading to disproportionate failures in downstream reasoning tasks.

Existing robustness evaluations primarily focus on aggregate performance metrics (e.g., average error rates or absolute robustness) under static corruption settings. These approaches fail to capture:

1. How reasoning stability degrades progressively as corruption severity increases.
2. How visual distortions propagate into specific multimodal reasoning failure modes (e.g., symbolic vs. semantic errors).
3. The impact of directional information loss (spatial traversal) on reasoning capabilities.

2. Methodology: The BRUCE Framework

The authors propose BRUCE (Benchmarking Robustness Under Corruption Escalation), a multimodal reasoning fragility framework designed to analyze VLM robustness across scientific (chemistry) and mathematical reasoning tasks. The framework operates through a six-step pipeline:

1. Clean Inputs and Data: Establishes baseline performance using clean image-question pairs from datasets like ScienceQA-Chemistry, MMJEE-Chemistry-MCQ, and MathVista-MCQ.
2. Perturbation Generation: Applies progressive perturbations at multiple severity levels. This includes single-condition corruptions (Gaussian blur, low contrast, rotation, center crop) and spatial traversal perturbations, where image content is progressively removed in specific directions (top-down, bottom-up, left-to-right, right-to-left).
3. Model Inference: The VLM performs inference on perturbed and traversal-modified samples, recording predictions, confidence scores, reasoning traces, and correctness signals.
4. Robustness Signal Computation: Calculates degradation signals including predictive accuracy loss, calibration drift (Expected Calibration Error), entropy collapse (reduction in prediction diversity), and failure propagation.
5. Metric Computation: Aggregates these signals into two novel metrics:
   • Robustness Corruption Index (RCI): Quantifies robustness degradation under increasing corruption severity by jointly modeling accuracy loss, calibration drift, entropy instability, and failure propagation.
   • Traversal-RCI (T-RCI): Extends RCI to progressive spatial traversal. It captures directional sensitivity, spatial evidence removal, and the dynamics of progressive reasoning collapse.
6. Failure Taxonomy Analysis: Categorizes prediction failures into four high-level domains:
   • OCR-dependent reasoning: Errors in text extraction/interpretation.
   • Spatial reasoning: Errors in geometric or structural understanding.
   • Symbolic reasoning: Errors in numerical, logical, or scientific process manipulation.
   • Semantic failures: Misinterpretation of contextual meaning despite partial visual understanding.

3. Key Contributions

The paper makes the following specific contributions:

• Framework Proposal: Introduction of BRUCE, a comprehensive framework for evaluating corruption-induced reasoning failures in scientific VLMs across chemistry and math benchmarks.
• Novel Metrics: Development of RCI and T-RCI to quantify the rate of performance deterioration and directional fragility, moving beyond static accuracy measurements.
• Progressive Perturbation Protocol: A controlled protocol spanning progressive blur, rotation, contrast degradation, and spatial cropping, including directional traversal to assess robustness under progressive information loss.
• Hierarchical Failure Taxonomy: A structured classification of reasoning failures (OCR, Spatial, Symbolic, Semantic) that enables interpretable analysis of how visual perturbations propagate into specific reasoning breakdowns.
• Empirical Evidence: Demonstration that high clean-task accuracy does not reliably predict robustness under corruption escalation, and that different perturbation families induce distinct failure modes.

4. Experimental Results

The framework was evaluated on seven VLMs (including GPT-4.1, Claude Opus 4.7, Gemini 3.5 Flash, Qwen2.5-VL-72B, InternVL2.5-8B, LLaVA-NeXT-Mistral-7B, and ChemVLM-8B) across three datasets. Key findings include:

• Dataset Dependency: Robustness varies significantly by dataset. ScienceQA-Chemistry showed high stability (low RCI), while MMJEE-Chemistry-MCQ remained highly sensitive to perturbations, with corrupted accuracies often below 25%.
• Model Performance:
  • InternVL2.5-8B and Qwen2.5-VL-72B consistently occupied the strongest robustness regimes, achieving low RCI/T-RCI scores and high retained accuracy.
  • GPT-4.1 showed high clean accuracy on MMJEE-Chemistry but exhibited the highest traversal degradation, suggesting a reliance on complete visual evidence rather than robustness to localized loss.
  • Claude Opus 4.7 demonstrated that high clean performance does not guarantee robustness, recording the highest mean RCI on ScienceQA-Chemistry.
  • ChemVLM-8B (a chemistry-specialized model) did not show a clear robustness advantage over general-purpose models, suggesting that domain specialization alone does not compensate for visual grounding deficiencies under corruption.
• Failure Modes: Across most models, symbolic reasoning failures dominated the error distribution under traversal perturbations. Models often retained visual perception but failed to manipulate the required numerical or logical relationships. GPT-4.1 was an exception, showing a higher proportion of semantic/contextual failures, indicating it maintained symbolic reasoning but struggled with global scene interpretation when regions were removed.
• Directional Sensitivity: Traversal analysis revealed that robustness is not uniform; specific directions (e.g., Right-to-Left) were consistently more fragile for certain datasets, indicating non-uniform distribution of critical visual evidence.

5. Significance and Claims

The paper claims that BRUCE provides a necessary shift in evaluating VLMs from static accuracy to dynamic robustness analysis. The authors argue that:

• Accuracy is Insufficient: High clean-task performance is not a reliable predictor of robustness under corruption escalation.
• Distinct Failure Mechanisms: Robustness to static corruption and robustness to progressive spatial information loss capture distinct failure mechanisms.
• Reasoning vs. Perception: Many robustness failures arise from reasoning collapse (specifically symbolic reasoning breakdown) rather than purely perceptual degradation.
• Diagnostic Utility: Traversal perturbations serve as a diagnostic tool to reveal which image regions are critical for reasoning, exposing "shortcut" reliance on localized cues rather than holistic understanding.

The work concludes that designing robust scientific VLMs requires addressing not just domain knowledge, but also visual grounding, multimodal alignment, and resilience to progressive information loss.