Technical Summary: Science Edge Evaluation (SEE)

Problem Statement

Current Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) are increasingly integrated into scientific workflows, yet their capacity to support complex, real-world laboratory science remains unverified. Existing evaluation paradigms suffer from three primary limitations:

1. Superficiality: They prioritize breadth over depth, relying on sanitized, exam-style problems that reward pattern matching and memorization rather than dynamic, evidence-driven reasoning.
2. Unimodal Bias: They often ignore the multimodal nature of scientific reasoning, which requires the joint interpretation of figures, spectra, microscopy images, numerical data, and text.
3. Disciplinary Silos: They treat scientific domains (chemistry, biology, materials science) as isolated, failing to capture the interdisciplinary integration required for modern research.

Consequently, current benchmarks do not accurately measure whether models can reason from incomplete, heterogeneous, and cross-disciplinary experimental evidence—a core requirement for authentic scientific discovery.

Methodology: Science Edge Evaluation (SEE)

To address these gaps, the authors introduce Science Edge Evaluation (SEE), a multimodal benchmark grounded in peer-reviewed literature and expert experimental practice.

Data Construction

• Sources: Questions are derived from real scientific research settings, including figures and data from peer-reviewed papers and first-hand experimental data contributed by active frontline experts (PhD holders or candidates) in chemistry, biology, and materials science.
• Scope: The benchmark comprises 1,116 questions (1,049 publicly released) spanning 17 fine-grained sub-fields across the three main disciplines. It includes both within-discipline and cross-disciplinary questions.
• Modality: Tasks are multimodal, requiring the interpretation of diverse visual evidence such as Cryo-EM structures, Western blots, NMR/MS spectra, SEM/TEM/AFM images, X-ray diffraction patterns, and thermal analysis curves.
• Quality Control: A rigorous pipeline ensures data integrity:
  1. Standardization: Uniform wording, terminology, and image resolution.
  2. AI-Assisted Checks: Automated verification of format and answer uniqueness.
  3. Expert Review: Domain experts verify scientific accuracy and label correctness.
  4. Image-Ablation Checking: Questions solvable by text alone are removed to ensure visual evidence is indispensable.

Evaluation Protocol

• Models Tested: 19 representative MLLMs were evaluated, including 15 general-purpose models (e.g., GPT-5.6-Sol, Gemini 3.1 Pro, Qwen3.8-Max) and 4 science-specialized models (e.g., Intern-S2 Preview, S1-VL).
• Scoring: A strict binary scoring protocol is used. For multiple-choice questions, answers must exactly match the ground truth. Numerical answers are evaluated with expert-defined tolerances. Refusals and non-answers are counted as incorrect.
• Judge Model: A "LLM-as-a-judge" pipeline (using Gemini 3.1 Pro) extracts and verifies answers, with robustness confirmed via alternative judges.

Experimental Variations

1. Modality Ablation: Images were removed to test if models recognize missing information or hallucinate answers based on text priors.
2. Tool-Augmented Visual-Agent: Top-performing models were tested with access to web search and code interpreters to evaluate if tool use can bridge the reasoning gap.

Key Results

1. Overall Performance is Low

Even the best-performing model, GPT-5.6-Sol (Max), achieved only 48.7% accuracy on the full dataset. No model exceeded 50%. The average accuracy across all 19 models was significantly lower, with general-purpose models averaging 34.9% and science-specialized models averaging 22.2%.

2. General-Purpose Models Outperform Specialized Ones

Contrary to the expectation that domain-specific training yields better results, general-purpose models outperformed science-specialized models on average. This suggests that specialized training improves familiarity with terminology but does not necessarily enhance the ability to integrate visual experimental evidence with specific contexts.

3. Interdisciplinary Reasoning is a Bottleneck

Performance degrades as the complexity of interdisciplinary integration increases:

• Single-discipline questions: 35.1% average accuracy.
• Cross-discipline questions: 28.7% average accuracy.
• Complexity Gradient: Accuracy drops from 36.2% (1 sub-discipline label) to 28.2% (4 sub-discipline labels), indicating models struggle to coordinate knowledge across multiple scientific concepts and evidence types.

4. Evidence Grounding and Hallucination

In the modality ablation study (text-only input), models identified missing visual information in only 4.6% of cases. Instead of abstaining, they frequently generated answers based on linguistic priors or training data patterns, demonstrating a lack of awareness regarding evidential boundaries.

5. Tool-Augmented Agents

Introducing web search and code interpreters improved the best model's accuracy from 48.7% to 52.7%. While tools expanded information access and allowed for code-assisted image analysis, they also introduced new failure modes:

• Beneficial: Fact retrieval (47.7%), code-assisted image analysis (41.6%).
• Detrimental: Evidence integration failures (40.0%), wrong action selection (26.9%), and endless overthinking (11.7%).
  The results indicate that tool access alone does not guarantee reliable reasoning; the ability to manage tool-derived information within the bounds of original evidence is the critical challenge.

Failure Analysis

The paper identifies specific failure modes preventing reliable scientific discovery:

• Perception: Models lack quantitative precision in reading visual data (e.g., extracting exact values from spectra) and often suppress conflicting visual input in favor of high-frequency training patterns.
• Inference:
  • Lack of Global Coherence: Models fail to integrate constraints across multiple subfigures or detect logical contradictions within options.
  • Overextrapolation: Models treat preliminary or isolated results as conclusive, inventing evidence to justify claims.
  • Coordination Bottlenecks: Tasks requiring simultaneous visual perception, domain knowledge, and logical reasoning (e.g., NMR interpretation) lead to multiplicative error compounding.

Significance and Claims

The paper claims that SEE reveals a fundamental limitation in current MLLMs: they cannot reliably make justified, evidence-bounded inferences from experimental results.

• The Gap: The primary barrier to scientifically useful AI is not a lack of knowledge, domain specialization, or tool access, but the inability to manage multimodal evidence and reason strictly within the boundaries of available data.
• The Shift: Scientific evaluation must evolve from testing knowledge recall to assessing evidence-grounded, multimodal, interdisciplinary, and uncertainty-aware reasoning.
• Future Direction: The authors argue that future benchmarks for scientific agents must evaluate the interaction trajectory—assessing whether actions are justified by evidence, whether uncertainty is recognized, and whether conclusions are revised when new observations contradict assumptions.

The paper concludes that bridging the gap between current AI capabilities and real scientific discovery requires models to transition from explaining established concepts to deriving novel, evidence-based insights from experimental data.