OmniMatBench: A Human-Calibrated Multimodal Reasoning Benchmark Across 19 Materials Science Subfields

Technical Summary of OmniMatBench

Problem Statement
While Multimodal Large Language Models (MLLMs) are increasingly expected to assist in scientific research, their reliability in domain-specific reasoning remains unclear. Materials science presents a particularly challenging testbed due to its interdisciplinary nature, reliance on multimodal data (text, images, charts, formulas), and strong engineering application focus. Existing benchmarks in this domain largely focus on property prediction, knowledge-based QA, visual recognition, or final-answer accuracy. However, these protocols often fail to systematically evaluate the full reasoning process required to move from materials knowledge to application, including structure–property interpretation, processing understanding, fabrication logic, and decision-making. There is a significant gap in evaluating how models handle formula selection, unit-aware computation, and structured execution across diverse materials subfields.

Methodology
To address these limitations, the authors introduce OmniMatBench, a human-calibrated multimodal reasoning benchmark. The dataset construction involved a multi-stage pipeline with collaboration from materials-science specialists:

• Data Collection: 3,171 expert-curated questions were extracted and adapted from classical and contemporary materials knowledge resources.
• Taxonomy: The benchmark spans 19 distinct materials-science subfields, organized into four overarching domains: Fundamental Materials Science, Structural and Engineering Materials, Materials Processing and Manufacturing, and Functional and Applied Materials. This structure reflects a Knowledge–Structure–Processing–Application (KSPA) perspective, including underrepresented but critical fields like gemology, powder materials, and welding technology.
• Task Types: The benchmark consists of two primary task types:
  1. Open-ended QA: Annotated with expert-curated key points covering concepts, reasoning elements, and scoring criteria.
  2. Calculation (CAL): Structured problems requiring specific answer slots, formula traces, unit requirements, numerical precision constraints, and output schemas.
• Evaluation Protocol: The evaluation goes beyond simple answer matching. QA responses are scored using Precision, Recall, and F1 based on key point coverage. CAL responses are evaluated using strict exact-slot accuracy, parsing answers into structured components (numbers, formulas, units).
• Model Evaluation: The authors evaluated 13 frontier MLLMs (8 closed-source and 5 open-source), including models like Claude Opus 4.7, GPT-5.5, and Gemini 3.1 Pro, under zero-shot, formula-assisted, retrieval-assisted, and code-execution settings.

Key Contributions

1. OmniMatBench Dataset: A comprehensive benchmark containing 3,171 expert-verified problems across 19 subfields and 4 domains, covering both foundational and engineering-oriented topics.
2. Fine-Grained Evaluation Design: A protocol that combines rubric-based QA scoring with strict slot-based CAL evaluation. This design enables the detection of failures in concept identification, formula selection, multimodal grounding, unit-aware computation, and answer formatting.
3. Systematic Analysis of MLLM Capabilities: A large-scale evaluation of 13 models that reveals specific failure modes, including fixed reasoning heuristics, uneven knowledge across subfields, and limitations in applying knowledge to execution even with external tools.

Results
The evaluation reveals a substantial gap between current MLLMs and reliable materials-science reasoning:

• Overall Performance: The best-performing model, Claude Opus 4.7, achieved an overall score of only 0.372 (a weighted average of QA Macro-F1 and CAL slot accuracy). GPT-5.5 scored 0.324, and most other models scored below 0.30.
• Subfield Variation: Performance varies significantly across the 19 subfields. Models perform relatively better on fundamental topics (e.g., metallic materials, materials chemistry) but struggle with specialized engineering scenarios (e.g., metallurgical engineering, nanomaterials, composite molding) that require procedural knowledge and application-oriented judgment.
• QA vs. CAL: While models often achieve higher Recall than Precision in open-ended QA (generating broad but sometimes unsupported content), they face a much stronger bottleneck in CAL tasks. The best CAL slot accuracy was only 30.10%, with most models falling below 25%, indicating a significant gap between scientific fluency and reliable scientific execution.
• Tool Assistance: Providing formula assistance (Oracle Context) improved performance but did not saturate it, suggesting the bottleneck lies in applying the correct solution plan rather than just accessing formulas. Code execution did not consistently improve performance; in some cases, it degraded results because models failed to translate the materials context into the correct computational procedure.

Significance
The paper claims that OmniMatBench provides crucial insights into the capabilities and limitations of current MLLMs in materials science. By highlighting the "knowledge-to-execution" gap, the benchmark establishes a foundation for developing more reliable AI assistants for materials-science research. The authors emphasize that current models, while capable of generating plausible explanations, still lack the precision required for specialized engineering scenarios, process-specific mechanisms, and structured scientific computation. The benchmark serves as a rigorous testbed to drive future improvements in scientific reasoning, moving beyond simple knowledge recall to complex, multimodal, and application-driven problem solving.