Technical Summary: MMArch

Problem Statement

Multimodal Large Language Models (MLLMs) have demonstrated proficiency in engineering imagery tasks such as drawing recognition, information extraction, and compliance checking. However, a critical gap remains in principle-grounded visual reasoning: the ability to combine distributed visual evidence from multiple figures with explicit engineering principles to reach a conclusion. Existing benchmarks in the Architecture, Engineering, and Construction (AEC) domain primarily test single-figure perception (e.g., parsing floor plans) or textual knowledge recall (e.g., building codes). None effectively isolate the joint requirement of selecting relevant visual evidence, connecting it through domain-specific principles, and deriving a problem-specific conclusion. Without a benchmark that enforces this tripartite reasoning, the professional reasoning competence of MLLMs cannot be reliably measured or improved.

Methodology

Benchmark Construction (MMArch)

The authors introduce MMArch, a benchmark comprising 1,212 short-answer items derived entirely from figures in peer-reviewed AEC literature (SCI-indexed journals and arXiv). The construction process employs a rigorous, decoupled pipeline designed to prevent hallucination and shortcut exploitation:

1. Data Source: Approximately 10,000 papers were screened across ten subdomains (Structural, Seismic, Building Physics, Inspection, Heritage, Spatial, Envelope, Urban, BIM, and Computational Design). Only figures presenting measurable or diagnostic content were retained.
2. Decoupled Planner–Writer Pipeline:
   • Planner: Selects an "evidence lead" (a specific passage discussing the figure) and distills a reference answer (frozen to ≤10 tokens) along with supporting visual observations.
   • Writer: Composes a free-form question around the frozen answer. This separation ensures the question cannot be tailored to the answer, and the answer cannot be biased by the question phrasing.
3. Quality Control & Validation:
   • Automated Screening: Candidates are tested against text-only and caption-only inputs to ensure the answer cannot be derived without visual evidence. Items solvable by text alone are discarded.
   • Blind Adversarial Audit: Three blind agents verify evidence visibility, the necessity of domain principles, and the absence of linguistic shortcuts.
   • Dual Verification: Answers are independently re-derived via two paths (source text span and image crop computation) and reconciled.
   • Expert Review: Three professional architects and engineers retain an item only via unanimous agreement.
4. Reasoning Stages: Each item is designed to require three functional stages: Perceive (extracting evidence), Know (identifying the governing principle), and Judge (applying the principle to the evidence).

Evaluation Setup

The benchmark was used to evaluate 18 MLLMs, including open-weight models (Qwen, Gemma, InternVL, Llama families, MiniMax M3) and proprietary systems (GPT-4o, GPT-5.5, Claude Opus 4.8, Gemini variants).

• Metrics: Accuracy (%) based on the match between the model's short-answer output and the reference answer, normalized for units and synonyms.
• Baseline: A panel of domain experts (uninvolved in construction) achieved a 94.57% average accuracy.
• Prompting Tests: The study evaluated the impact of Chain-of-Thought (CoT) and visual panel identifiers on performance.

Key Results

Performance Gap

A significant performance gap exists between current MLLMs and human experts:

• Human Panel: 94.57% accuracy.
• Best Proprietary Model (GPT-5.5): 51.73% accuracy.
• Best Open-Source Model (MiniMax M3): 29.94% accuracy.
• Other Models: Most open-source models scored between 13% and 26%.

The gap of over 40 percentage points indicates that current models struggle fundamentally with the joint reasoning required, rather than simply lacking visual recognition capabilities.

Scaling and Prompting

• Model Scaling: Increasing parameter counts yielded limited gains. For example, scaling Llama-3.2 from 11B to 90B improved performance by only ~4 points. This suggests the bottleneck is not raw capacity but the ability to bind visual evidence to domain knowledge.
• Prompting Interventions:
  • Chain-of-Thought (CoT): Yielded inconsistent results, providing a minor gain for GPT-4o (+1.5%) but degrading performance for other models (e.g., -3.1% for Qwen3.5-35B-A3B).
  • Visual Prompting (Panel Identifiers): Had mixed effects, improving some models slightly while degrading others.
  • Conclusion: The limitations are not due to a lack of explicit reasoning traces or visual alignment ambiguity, but rather a deficit in domain-specific knowledge and compositional reasoning.

Error Analysis

Manual inspection of failures revealed five error categories:

1. Composition Error (35.8%): The dominant failure mode. Models correctly extract intermediate quantities but fail to combine them correctly (e.g., omitting a term in a calculation).
2. Principle Error (21.7%): Correct evidence reading but application of an intuitive heuristic instead of the governing engineering principle.
3. Perception Error (20.3%): Misreading visual signals (e.g., confusing overlapping curves).
4. Grounding Error (14.5%): Correct identification of a quantity but misreading its value or sign.
5. Consistency Error (7.7%): Internal analysis contradicts the final stated answer.

The distribution indicates that combining evidence and principles is the primary bottleneck, rather than locating the evidence.

Significance and Claims

The paper positions MMArch as a diagnostic tool to measure principle-grounded multimodal reasoning in the AEC domain. Its significance lies in:

1. Isolating the Reasoning Gap: By strictly filtering out text-only and single-figure shortcuts, MMArch proves that the current underperformance of MLLMs is due to an inability to synthesize visual evidence with engineering principles, not just a lack of visual literacy.
2. Rigorous Construction: The decoupled planner-writer pipeline and multi-stage validation (including blind adversarial audits and unanimous expert review) set a new standard for benchmark quality, ensuring ground truth is robust against hallucination.
3. Direction for Future Research: The results suggest that future progress in professional AI will require advances in model architecture or training to better integrate domain knowledge with visual reasoning, rather than relying on prompting techniques or simple parameter scaling. The paper explicitly states it does not probe field imagery or models beyond a current snapshot, serving instead as a controlled testbed for these specific reasoning capabilities.