Technical Summary: InteractScience

Problem Definition

The paper addresses a critical gap in the evaluation of Large Language Models (LLMs) regarding Interactive Scientific Demonstration Code Generation. While current benchmarks excel at static code generation (e.g., creating a blog webpage) or knowledge-based reasoning (e.g., explaining Newton's laws), they fail to assess the synthesis of deep domain knowledge with precise, reactive coding required for interactive scientific applications.

Existing evaluation methodologies are insufficient because they typically bifurcate into:

1. Static Code Generation: Focusing on layout or functional UI elements without verifying scientific logic.
2. Knowledge Question Answering: Assessing textual explanations without testing interactive behavior.
3. Visual-Only Judging: Relying on Vision-Language Models (VLMs) without reference snapshots or checklists, leading to subjective judgments that may miss scientific inaccuracies.

The core challenge is that models often generate code that is functionally interactive (buttons click, sliders move) but scientifically incorrect (physics laws are violated, visualizations do not match the underlying principles). Current benchmarks lack mechanisms to verify whether user actions correctly trigger intended scientific behaviors.

Methodology

To address these limitations, the authors introduce InteractScience, the first benchmark dedicated to the holistic creation of interactive scientific applications. The methodology is built on a hybrid evaluation framework combining two complementary components:

1. Programmatic Functional Testing (PFT)

PFT provides deterministic verification of the interaction logic defined in an implementation plan.

• Mechanism: It uses Playwright to execute a sequence of simulated user actions (e.g., clicking a button, dragging a slider) and asserts specific DOM states.
• Goal: To ensure that the generated code behaves exactly as specified in the implementation plan, verifying event handling, state updates, and component existence.
• Output: Binary pass/fail metrics (Overall Pass Rate, Average Pass Rate, Perfect Pass Rate).

2. Visually-Grounded Qualitative Testing (VQT)

VQT evaluates the correctness of the visualization and its alignment with scientific principles.

• Mechanism: It leverages reference snapshots (ground truth from the Wolfram Demonstrations Project) and checklists derived from scientific principles.
• Process: The framework executes actions to reproduce a specific state, captures a screenshot, and compares it against the reference snapshot.
• Scoring:
  • Perceptual Similarity: Measured via CLIP scores between the generated and reference snapshots.
  • Semantic Correctness: A VLM-as-Judge (Gemini-2.5-Pro) evaluates the generated snapshot against the reference and a specific checklist of inspection points (e.g., accuracy of numerical values, alignment of graphical elements, correctness of labels).
• Goal: To detect scientific errors that might pass DOM assertions but render incorrect phenomena (e.g., a spring-mass system vibrating with the wrong frequency).

Benchmark Construction

• Source Data: 150 interactive demonstrations from the Wolfram Demonstrations Project across five disciplines: Mathematics, Physics, Chemistry, Earth Science, and Computer Science.
• Synthesis: Using Gemini-2.5-Pro, the authors synthesized implementation plans, PFT test cases/scripts, VQT test cases/scripts, and evaluation checklists.
• Validation: A manual validation experiment involving graduate students confirmed the faithfulness and correctness of the synthesized suites (average scores > 4.0/5.0).
• Scale: The benchmark includes 150 problems, 779 PFT unit tests, and 590 VQT unit tests, totaling over 1,300 individual tests.

Key Contributions

1. Hybrid Evaluation Framework: The design of a unified protocol that jointly validates interaction logic (via PFT) and visually-grounded scientific fidelity (via VQT), moving beyond static or purely textual evaluation.
2. InteractScience Benchmark: The release of a rigorous evaluation suite featuring unit test scripts, reference snapshots, and checklist-guided judging for 150 diverse scientific problems.
3. Large-Scale Empirical Analysis: A systematic evaluation of 30 state-of-the-art LLMs (both open- and closed-source) revealing specific limitations in integrating scientific reasoning with interactive code generation.

Experimental Results

The evaluation of 30 models (including GPT-5, Claude-Sonnet-4, DeepSeek-R1, and Qwen3 series) yielded the following insights:

• Overall Performance: The benchmark is highly challenging. The Perfect Pass Rate (PPR) for PFT rarely exceeds 16%, indicating that few models can flawlessly follow complex implementation plans.
• The "Superficial Competence" Gap:
  • Action Success Rate (ASR) is consistently high (>85%), showing models can generate functional UI components (buttons, sliders) that execute without error.
  • Semantic Correctness (VLM-Judge) is significantly lower (often <60), revealing that while the interface works, the underlying scientific logic is frequently incorrect.
• Model Comparisons:
  • Closed-source models (e.g., Claude-Sonnet-4, GPT-5) generally outperform open-source models, particularly in functional correctness.
  • Model Scale: Larger models (e.g., DeepSeek-R1-0528, Qwen3-235B) perform comparably to mid-tier proprietary models, while smaller models (<14B) struggle significantly.
• Complementarity of Metrics: Spearman correlations between PFT and VQT metrics are low (<0.32), confirming that functional correctness and visual/scientific fidelity are distinct capabilities. A model can pass functional tests but fail scientific visualization, and vice versa.
• Impact of Reference Snapshots: Providing reference snapshots as input to multimodal models yields modest improvements in visual style and layout but does not fully resolve the bottleneck of grounding depicted states in correct scientific semantics.

Significance and Claims

The paper claims that InteractScience establishes a crucial foundation for developing reliable AI-driven tools in science and education. Its significance lies in:

1. Standardization: It provides the first unified protocol to evaluate the combined capability of scientific reasoning and interactive coding, a task previously unassessed by existing benchmarks.
2. Diagnostic Capability: By exposing the gap between "superficial competence" (working UI) and "scientific fidelity" (correct logic), the benchmark helps identify specific failure modes (e.g., logic errors, numerical parameter errors) in current models.
3. Auditing Tool: The authors position InteractScience as an auditing tool to identify risks where AI-generated materials might appear visually plausible but contain scientifically incorrect explanations, emphasizing that benchmark scores should not be interpreted as certification of scientific correctness without expert review.

The work concludes that while LLMs show promise in automating software creation, significant gaps remain in their ability to synthesize deep domain knowledge into functional, interactive scientific demonstrations. Future work is suggested to expand the benchmark, incorporate broader expert validation, and explore agent-driven testing.