Technical Summary: SciMIF

Problem Statement

The application of Multimodal Large Language Models (MLLMs) in scientific domains has evolved from basic question-answering to complex paradigms like autonomous scientific agents. However, current evaluation methods primarily focus on scientific correctness (whether the final answer is factually accurate) rather than instruction adherence (whether the model satisfies explicit operational constraints).

Existing general-purpose instruction-following benchmarks (e.g., IF-Eval, FollowBench) assess constraints on format, length, and style but lack domain-specific scientific constraints. Conversely, existing scientific benchmarks (e.g., SciBench, MMMU) evaluate reasoning and knowledge but do not systematically test a model's ability to follow complex, multi-step operational instructions. This gap obscures two distinct capabilities: a model may produce a scientifically correct answer that violates requested units or formats, or it may strictly follow a format while relying on invalid scientific reasoning. Furthermore, scientific instruction following is uniquely challenging due to its dependence on domain-specific knowledge, interdisciplinary variations in semantics, and frequent reliance on multimodal inputs (e.g., molecular structures, microscopy images).

Methodology

1. Taxonomy of Scientific Constraints

The authors introduce SciMIF (Scientific Multimodal Instruction Following), a benchmark designed to evaluate MLLMs across five scientific disciplines: Chemistry, Geography, Biology, Materials Science, and Physics.

Based on an analysis of 22 distinct tasks across these disciplines, the authors constructed a hierarchical taxonomy comprising:

• 10 Functional Constraint Groups: These capture shared requirements across domains, including Procedure, Number, Method, Unit, Format, Terminology, Precision, Letter, Structure, and Selection.
• Discipline-Specific Instantiations: While the functional groups are shared, their concrete meanings vary. For example, a "Terminology" constraint requires valid molecular nomenclature in Chemistry, hierarchical addresses in Geography, and characterization techniques in Materials Science.
• Constraint Inventory: The final benchmark includes 42 distinct constraints derived from the 10 groups, adapted to the conventions of each discipline.

2. Data Construction Pipeline

SciMIF is constructed by systematically augmenting 13 existing scientific datasets (totaling 2,527 samples) through a four-step pipeline:

1. Seed Preparation: Selecting samples with a question ($q$), optional visual input ($I$), reference answer ($a$), and task type ($t$).
2. Constraint Recognition: Identifying constraints already implicit in the original query to avoid redundancy.
3. Constraint Injection:
   • Scientific Constraints: Injecting domain-specific constraints compatible with the task type.
   • General Constraints: Injecting general operational constraints (e.g., output format, capitalization) without altering the scientific correctness of the reference answer.
   • Validation: An automatic double-check ensures the injected constraint is present in the query and that the ground-truth answer remains valid.
4. Human Verification: Two annotators review samples for logical coherence, fluency, and constraint fidelity. Problematic samples are revised or discarded.

3. Evaluation Metrics

The benchmark employs three metrics to assess performance:

• Constraint Satisfaction Rate (CSR): The average proportion of satisfied constraints across all instructions.
• Instruction Satisfaction Rate (ISR): The proportion of instructions where all associated constraints are completely satisfied.
• Decomposed Requirements Following Ratio (DRFR): Measures compliance at the level of individual decomposed requirements across the total number of constraints.

Verification utilizes script-based methods (for deterministic checks like format and units) and LLM-as-a-Judge protocols (for semantic checks like reasoning steps).

Key Results

1. Performance Disparities Across Disciplines

Evaluation of state-of-the-art closed-source (e.g., GPT-5.2, Claude-Sonnet-4.6) and open-source models (e.g., Qwen3.5, InternVL3.5) reveals significant variation:

• Chemistry poses the greatest challenge, with the highest-performing model (GPT-5.2) achieving an ISR of only 46.33%.
• Biology and Materials Science show higher performance (ISR ~72–78%).
• Geography also presents significant difficulties due to spatial hierarchies.
• Physics generally yields the highest DRFR scores (92.2% average).

2. Model Scale and Architecture

• Scaling Paradox: Increasing model parameters does not yield linear improvements in instruction adherence. For instance, in the Qwen3.5 series, the 122B model performed slightly worse (50.41% ISR) than the 27B model (51.37% ISR).
• Closed vs. Open Source: Closed-source models consistently outperform open-source counterparts across all disciplines, suggesting advantages in data quality and alignment strategies.

3. Constraint Domain Analysis

• General vs. Scientific Constraints: Models perform significantly better on scientific constraints (semantically coupled to the task) than on general constraints (formatting/structure). For GPT-5.2, DRFR on scientific constraints was 88.74% versus 74.65% on general constraints.
• Fine-Grained Challenges: Models struggle most with Letter and Number constraints, which require precise symbolic processing and domain-specific entity recognition (e.g., counting chemical bonds vs. counting characters).

4. Correctness vs. Adherence

A critical finding is the weak coupling between scientific correctness and instruction adherence:

• Only ~30% of samples achieved both scientific correctness and full instruction adherence (Correct and Followed).
• Approximately 20% of samples were scientifically correct but violated constraints (Correct but Violated).
• Over 30% of samples followed instructions but produced incorrect scientific answers (Incorrect but Followed).
• Statistical analysis (Pearson $\chi^2$ test) confirms that while correctness and adherence are positively associated, the strength of this association is modest ($\phi$ coefficients ranging from 0.06 to 0.20), indicating they are distinct capabilities.

Significance and Contributions

The paper claims three primary contributions:

1. Expert-Derived Taxonomy: The establishment of a comprehensive taxonomy of scientific constraints spanning five disciplines and ten functional groups, capturing both shared capabilities and domain-specific requirements.
2. Scalable Framework: A methodology for transforming existing scientific tasks into instruction-following evaluations by recognizing implicit constraints and injecting compatible constraints without altering reference answers.
3. Benchmark and Evaluation: The construction of SciMIF and the evaluation of representative MLLMs, which reveals:
   • Substantial disciplinary variation in performance.
   • Severe difficulties with fine-grained constraints.
   • A clear gap between scientific correctness and instruction adherence.

The authors conclude that current MLLMs are not yet reliable for rigorous scientific applications requiring strict operational constraints. They suggest future research should focus on domain-aware instruction alignment, integrating external tools (e.g., symbolic engines) for fine-grained constraints, and optimizing for joint attainment of correctness and adherence rather than treating them in isolation.