How Far Are We? Systematic Evaluation of LLMs vs. Human Experts in Mathematical Contest in Modeling

1. Problem Statement

While Large Language Models (LLMs) have demonstrated strong performance on isolated reasoning benchmarks (e.g., math problems, code generation), their ability to solve real-world, end-to-end problems remains unclear. Mathematical modeling competitions require a coherent workflow integrating problem identification, assumption formulation, model construction, computational implementation, and result validation.

Existing evaluation methods for LLMs in this domain suffer from two main limitations:

1. Problem-Agnostic Rubrics: They use static, generic criteria that fail to account for the specific objectives, constraints, and logical consistency of individual problems.
2. Report-Level Focus: They often reward superficial presentation quality or "plausible-sounding" text rather than the actual correctness, executability, and validation of the solution.

This creates a gap where LLMs may generate high-scoring reports that are fundamentally flawed, unexecutable, or misaligned with the problem requirements. The authors ask: How close are current LLMs to human expert performance in full-process mathematical modeling, and where do they fail?

2. Methodology: Problem-Oriented Stage-Wise Evaluation Framework

To address these limitations, the authors propose a novel evaluation framework grounded in problem semantics and modeling stages.

A. Framework Design

The framework operates in two phases with expert-in-the-loop verification:

1. Subtask Decomposition: A complex modeling problem is decomposed into distinct, self-contained subtasks (e.g., "Measurement Preprocessing," "Dynamic Model Design"). Domain experts verify that these subtasks collectively capture all original problem requirements.
2. Stage-Wise Criteria Instantiation: For each subtask, evaluation criteria are dynamically generated for seven canonical modeling stages:
   • Problem Identification
   • Problem Formulation
   • Assumption Development
   • Model Construction
   • Model Solving
   • Code Implementation
   • Result Analysis
   • Mechanism: An LLM generates specific, checkable criteria based on the subtask's inputs/outputs, which are then refined and verified by human experts. This ensures criteria are task-specific and not generic.

B. Dataset and Models

• Dataset: 97 problems from the China Postgraduate Mathematical Contest in Modeling (PMCM), representing expert-level difficulty (gold medal rate ~1.0–1.5%). Problems were converted from PDF to LaTeX for machine processing.
• Models Evaluated: A diverse set of state-of-the-art LLMs, including:
  • Qwen Family: 7B to 235B parameters (Qwen2.5 and Qwen3).
  • DeepSeek: V3.2-Instruct and V3.2-Thinking variants.
  • OpenAI: o4-mini.
• Protocol: All models used a unified agent framework to generate reports, ensuring that the only variable was the model itself.

C. Validation

The framework's reliability was validated by comparing automatic scores against independent judgments from human experts (medal-winning students). The study used the Intraclass Correlation Coefficient (ICC) to measure agreement.

3. Key Contributions

1. Novel Evaluation Framework: Introduced a problem-oriented, stage-wise evaluation system that replaces static rubrics with dynamically instantiated, expert-verified criteria. This captures global consistency across the modeling workflow.
2. Discovery of the "Comprehension–Execution Gap": Revealed a systematic performance disparity where LLMs excel at understanding and framing problems but fail significantly in execution-oriented stages (solving, coding, validation).
3. Evidence on Scaling Laws: Demonstrated that simply increasing model scale (e.g., from 7B to 235B parameters) improves comprehension but fails to close the execution gap, suggesting that scaling alone is insufficient for complex problem solving.
4. Failure Analysis: Identified specific failure patterns, primarily insufficient specification, missing verification, and lack of validation, which propagate through the pipeline without correction.

4. Key Results

A. Framework Reliability

• Alignment with Experts: The proposed framework achieved an ICC of 0.673 with human expert judgments, significantly outperforming the baseline (generic rubrics) which scored 0.012.
• Discriminative Power: Unlike baseline scores which clustered in a narrow high range (6.5–8.5), the proposed framework showed wide score dispersion, effectively distinguishing between high-quality and flawed solutions.

B. Performance Analysis (The Comprehension–Execution Gap)

• Early Stages (High Performance): LLMs scored near-expert levels (8–9/10) in Problem Identification and Formulation. They can effectively interpret problem statements and restate objectives.
• Middle Stages (Intermediate Performance): Scores dropped to 4–6/10 in Assumption Development and Model Construction. While the high-level intent was reasonable, the mathematical justification and derivation were often incomplete.
• Late Stages (Critical Failure): Performance collapsed in Model Solving, Code Implementation, and Result Analysis, with scores typically falling to 2–3/10.
  • Code Implementation: Often resulted in non-executable code or missing scripts.
  • Result Analysis: Lacked validation, sensitivity analysis, or comparison with baselines.

C. Impact of Model Scaling

• Comprehension: Scaling from 7B to 72B parameters improved early-stage scores, but gains diminished at larger scales.
• Execution: Scaling from 7B to 235B showed no consistent improvement in execution-oriented stages. The gap between understanding and doing persisted even in the largest models.

D. Failure Modes

Analysis of low-scoring cases revealed that failures were rarely due to "wrong ideas" but rather execution deficiencies:

• Assumptions: Stated but not checked or justified.
• Model Construction: Missing derivations or unclear variable definitions.
• Solving: Solutions proposed but not verified for stability or convergence.
• Propagation: Errors introduced in early stages (e.g., undefined assumptions) were carried forward without correction, leading to downstream failures.

5. Significance and Implications

• Rethinking LLM Evaluation: The paper argues that current benchmarks focusing on isolated tasks or report aesthetics are insufficient for evaluating real-world problem-solving. Evaluation must be process-aware and stage-specific.
• Beyond Scaling: The findings challenge the notion that "bigger is better" for complex workflows. Bridging the comprehension–execution gap requires new paradigms such as:
  • Process-aware reasoning: Models that explicitly track state and constraints across stages.
  • Self-correction mechanisms: Systems capable of revisiting and validating earlier steps (e.g., verifying assumptions before solving).
  • Tool integration: Tighter coupling with code execution environments and symbolic solvers to ensure reproducibility.
• Practical Application: The framework provides a rigorous method for assessing LLMs in scientific and engineering domains, highlighting that while LLMs are powerful assistants for ideation, they currently lack the procedural rigor required for autonomous expert-level problem solving.