Benchmarking Cross-Scale Perception Ability of Large Multimodal Models in Material Science

This paper introduces CSMBench, a specialized dataset of 1,041 material science figures spanning atomic to macro scales, to benchmark and reveal the significant limitations of current Large Multimodal Models in interpreting hierarchical structure-property relationships across diverse physical dimensions.

The Gist

Imagine you are trying to teach a super-smart robot how to understand the world of materials science. This isn't just about knowing what a rock or a metal is; it's about understanding how the invisible arrangement of tiny atoms creates the strength of a bridge or the speed of a computer chip.

The problem is that current AI models are like generalist tourists. They can look at a picture of a sunset and say, "Oh, that's a pretty orange sky." But if you show them a microscopic image of a metal grain or a diagram of atomic defects, they often get lost. They can't connect the tiny dots to the big picture.

This paper introduces a new "driving test" for these AI robots called CSMBench. Here is a simple breakdown of what they did and what they found, using some everyday analogies.

1. The Problem: The "Zoom" Gap

Materials science happens on four different "zoom levels," like a camera with different lenses:

• Atomic Scale (The Microscope): Looking at individual atoms and how they stick together.
• Micro Scale (The Magnifying Glass): Looking at tiny grains and cracks in a material.
• Meso Scale (The Hand Lens): Looking at larger structures, like the texture of a piece of wood or metal.
• Macro Scale (The Naked Eye): Looking at the final object, like a car engine or a building beam.

Existing AI benchmarks were like asking a tourist to identify a "tree." But in materials science, you need to know how the tree's roots (atoms) affect the height of the tree (the macro object). Current AI models are bad at zooming in and out to see the whole story.

2. The Solution: CSMBench (The New Driving Test)

The researchers built a new dataset called CSMBench. Think of it as a gym for AI vision.

• The Workout: They collected 1,041 high-quality images from top scientific journals (up to September 2025). Because these images are so new, the AI couldn't have memorized the answers from its training data. It had to actually think.
• The Exercises:
  1. The Essay Test: Show the AI an image and ask it to write a detailed explanation. (Can it describe what it sees in plain English?)
  2. The Multiple Choice Test: Show an image and four possible captions. Three are tricky "fake" answers (e.g., swapping "Gold" for "Silver" or changing a temperature from 100°C to 90°C). The AI has to pick the right one.

3. The Results: The AI's Struggles

When they ran the top AI models through this gym, here is what happened:

• The "Rich" vs. The "Open" Models: The big, proprietary models (like GPT-5.1 and Gemini) generally did better than the open-source ones. They were like experienced mechanics who could explain why a part failed, not just point to it. The open-source models were like apprentices—they could spot the broken part, but sometimes struggled to explain the physics behind it.
• The "Zoom" Problem: The AI models were surprisingly inconsistent. They were great at the Micro and Meso scales (the middle ground), where images look like standard photos. But they stumbled at the Atomic scale (too abstract) and the Macro scale (too complex). It's like a photographer who is great at taking portraits but terrible at taking landscapes or extreme close-ups.
• Bigger isn't Always Better: You might think a bigger AI brain (more "parameters") would be smarter. But the study found that size doesn't equal smarts here. A slightly smaller, newer model (Qwen3) actually beat a much larger, older model (Qwen2.5) because the new one had better "thinking processes" (reasoning), not just a bigger memory.
• The "Guessing" Trap: Some models were amazing at the Multiple Choice test (95% accuracy!) but terrible at the Essay test. This suggests they were cheating. They were good at spotting visual patterns to guess the right answer but couldn't actually write a coherent scientific story. It's like a student who can pick the right answer on a test by elimination but can't explain the lesson to a friend.

4. The Big Takeaway

The paper concludes that while AI is getting better at "seeing" pictures, it still struggles to understand the physics behind them.

• The Analogy: Current AI is like a tourist with a camera. They can take a great photo of a bridge and say, "That's a bridge." But they don't understand the tension in the cables, the chemistry of the steel, or why the bridge won't collapse in a storm.
• The Future: To truly help scientists discover new materials, AI needs to stop just "recognizing" images and start reasoning about them. CSMBench is the tool designed to push AI from being a "tourist" to becoming a "scientist."

In short: We built a tough new test to see if AI can understand materials science. The results show that while AI is getting smarter, it still needs to learn how to connect the tiny dots to the big picture before it can truly help us invent the future.

Technical Summary

1. Problem Statement

Materials science is fundamentally defined by hierarchical structure-property relationships, where macroscopic performance emerges from interactions across multiple physical scales (atomic, micro, meso, and macro). While Large Multimodal Models (LMMs) have advanced in general visual understanding, existing scientific benchmarks (e.g., ScienceQA, MMSCI) fail to address the specific challenges of this domain.

• Limitations of Current Benchmarks: Most rely on general chart interpretation, textbook knowledge, or common-sense reasoning. They do not evaluate a model's ability to traverse physical dimensions from angstrom-level lattices to centimeter-scale bulk materials.
• The Gap: There is a lack of rigorous evaluation for cross-scale reasoning, specifically the ability to interpret visual features that dictate material properties at different scales and link microscopic observations to macroscopic outcomes.

2. Methodology: CSMBench

The authors introduce CSMBench, a specialized benchmark designed to evaluate the hierarchical understanding of LMMs in materials science.

A. Data Collection & Curation

• Source: 1,041 high-quality figures curated from 432 peer-reviewed articles published up to September 2025 in top-tier journals (e.g., Nature, Science Advances, Advanced Materials). This ensures the data is "contamination-free" relative to current model training cutoffs.
• Scale Categorization: The dataset is strictly categorized into four physical regimes based on materials informatics principles:
  1. Atomic Scale ($10^{-10}$ m): Lattice arrangements, defects, interfaces.
  2. Micro Scale ($10^{-9}$ m): Nanometer-to-micrometer features (precipitates, dislocations).
  3. Meso Scale ($10^{-6}$ m): Micrometer-to-millimeter structures (grain boundaries, textures).
  4. Macro Scale (cm to m): Bulk materials, component geometries, deformation patterns.
• Preprocessing: Raw PDFs were converted to machine-readable formats (MinerU), followed by regex matching for figure references. A human expert filtering process removed generic flowcharts, retaining only images providing direct insights into morphology, phase, or composition.

B. Task Design

CSMBench employs two distinct evaluation tasks to measure both recognition precision and reasoning depth:

1. Open-Ended Figure Description:
   • Input: Image + Caption.
   • Task: Generate a 100–300 word scientific explanation focusing on visible features and deriving conclusions.
   • Evaluation: Measured via BERTScore (F1), Semantic Textual Similarity (STS), and an LLM-as-a-Judge score (1–10 scale).
2. Multiple-Choice Caption Matching:
   • Input: Image + 4 Options (1 correct, 3 distractors).
   • Task: Select the correct caption.
   • Distractor Generation: Three types of challenging negative distractors were generated using GPT-4.1:
     • Characterization Methods: Swapping techniques (e.g., TEM $\to$ SEM).
     • Material Compositions: Substituting plausible chemical formulas (e.g., $ZrB_2$ $\to$ $TiB_2$).
     • Numerical Values: Perturbing critical parameters (e.g., temperature, time).

3. Key Contributions

1. Novel Dataset: Creation of the first contamination-free, cross-scale material science dataset (1,041 figures) covering four distinct physical regimes.
2. Dual-Task Framework: Introduction of a hybrid evaluation protocol combining open-ended generation (testing reasoning) and multiple-choice matching (testing fine-grained discrimination).
3. Comprehensive Benchmarking: Evaluation of 10 state-of-the-art LMMs (both proprietary and open-source), revealing critical performance disparities across scales.

4. Key Results & Findings

The evaluation of models (including GPT-5.1, Gemini-2.5-Pro, Qwen2.5/3-VL, and InternVL3) yielded five major observations:

• Proprietary vs. Open-Source: Proprietary models (e.g., GPT-5.1, Doubao-1.6-vision) significantly outperform open-source models.
  • Open-Ended: GPT-5.1 achieved the highest LLM-as-a-Judge score (~8.07), while the best open-source model (Qwen2.5-VL-72B) trailed by ~1.8 points.
  • Multiple-Choice: Doubao-1.6-vision reached 95.59% accuracy, whereas open-source models generally hovered between 68–88%.
• Scale-Dependent Performance: Models perform better at Micro and Meso scales (standardized experimental images like SEM/CT) compared to Atomic and Macro scales.
  • Reason: Atomic and Macro scales contain more diverse schematic diagrams and heterogeneous visual representations, increasing interpretive complexity.
  • Correlation: While LLM-as-a-Judge scores showed high cross-scale consistency (>0.97), STS scores dropped significantly (to ~0.66), indicating that semantic precision is highly sensitive to the specific scale.
• Scaling Laws are Inconsistent: Simply increasing parameter count does not guarantee better scientific reasoning.
  • Example: In the Qwen2.5-VL family, the 72B model did not outperform the 7B model in open-ended descriptions.
  • Architecture Matters: Qwen3-VL-8B (with internal thinking processes) outperformed the much larger Qwen2.5-VL-72B (88.02% vs. 80.48% in matching), suggesting architectural evolution is more critical than raw scale for scientific tasks.
• Recognition vs. Reasoning Gap: High accuracy in multiple-choice tasks does not guarantee deep reasoning.
  • Example: Doubao-1.6-vision had the highest matching accuracy (95.59%) but was surpassed by GPT-5.1 in open-ended descriptions. This suggests some models rely on visual heuristics for selection but lack the ability to articulate physical principles.
• Heterogeneity Boosts Performance: Models performed better on hybrid images (morphology + statistical plots) than pure morphological images. The auxiliary statistical data acts as a contextual anchor, aiding disambiguation.

5. Significance

• Scientific Discovery: CSMBench establishes a rigorous standard for evaluating whether LMMs can truly act as scientific assistants capable of "cross-scale reasoning," a prerequisite for autonomous materials discovery.
• Model Development: The results highlight that current "generalist" models struggle with domain-specific physical laws. Future models require specialized training on hierarchical visual data and architectural improvements (e.g., internal reasoning chains) rather than just parameter scaling.
• Community Resource: By releasing the dataset and benchmark, the authors provide a critical tool for the AI-for-Science community to measure progress in bridging the gap between visual perception and deep physical understanding.

Availability: The CSMBench dataset is publicly available at: https://huggingface.co/datasets/lututu/CSMBench.