BEAMS: Benchmarking and Evaluating AI for Modeling and Simulation

Technical Summary: BEAMS – Benchmarking and Evaluating AI for Modeling and Simulation

Problem Statement
The proliferation of Large Language Models (LLMs) in modeling and simulation offers potential efficiency gains but introduces significant risks regarding opacity, lack of interpretability, and the inability to verify reasoning. Current AI tools often fail to construct explicit simulation models, which are necessary to apply established analytical methods for understanding how an AI arrives at conclusions. Without transparent, verifiable models, AI tools are inadequate for informing real-world decisions on complex societal problems. Furthermore, there is a lack of generalizable guidelines to evaluate whether AI tools complement human expertise or merely replace it, and no standardized benchmarks exist to drive the development of responsible, ethical AI for this domain.

Methodology
The BEAMS Initiative addresses these gaps by establishing an open digital and organizational infrastructure to collaboratively evaluate AI tools.

• Infrastructure: The initiative utilizes the open-source sd-ai project (hosted on GitHub) to provide a reusable platform. This platform includes a request handler that interfaces between a modeler's software client and external AI "engines." It enforces structured communication via parseable JSON outputs, ensuring responses are context-appropriate and machine-readable.
• Organizational Structure: The initiative operates through two working groups: a Steering Group (prioritizing benchmarks and principles) and a Technical Group (implementing automated tests). The initiative is anchored at the University at Buffalo but includes industry and academic collaborators.
• Design Principles: The evaluation framework is guided by eight principles, including "Do no harm," "Complement human abilities, don't substitute for the human," and "Deliver high quality models."
• Evaluation Framework: Tests are categorized by the stage of the modeling process (problem definition, conceptualization, formulation, analysis, policy analysis) and the type of engine (Qualitative, Quantitative, Discussion). The evaluation suite includes seven distinct categories:
  1. Causal Translation: Assesses the ability to transform plain-language descriptions into structured models using synthetic "fake alternate universes" with deterministic ground truth to isolate structural understanding from domain knowledge.
  2. Model Iteration: Tests the ability to add new causal relationships to existing models without breaking the original structure, using synthetic variables.
  3. Causal Reasoning: Evaluates the identification of expert-validated causal mechanisms in real-world scenarios (e.g., pandemic dynamics) rather than surface-level pattern matching.
  4. Conformance: Measures adherence to explicit user constraints (e.g., specific variables, loop counts) rather than matching a single ground truth.
  5. Model Behavior Explanation: Assesses the ability to explain simulation dynamics and feedback loop dominance based on provided JSON-encoded model data.
  6. Suggested Model Building Steps: Tests the ability to outline logical construction steps (stocks, flows, parameters) for a given problem.
  7. Suggested Model Fixes: Evaluates the ability to diagnose and explain formulation errors in existing models.

Key Contributions

• Open Benchmarking Framework: The creation of the sd-ai project provides the first open-source, vendor-agnostic infrastructure for testing AI modeling tools against standardized, automated benchmarks.
• Comprehensive Test Suite: The implementation of 116 total tests across three engine types (53 for Qualitative, 46 for Quantitative, 17 for Discussion) covering diverse modeling tasks.
• Principled Evaluation: The explicit alignment of technical tests with ethical and operational design principles (e.g., ensuring tools do not bypass the human modeling process).
• Empirical Data on LLM Performance: The generation of performance leaderboards that decouple engine performance from specific LLMs, allowing for the swapping of underlying models to test flexibility.

Results
The evaluation of various LLMs coupled with sd-ai engines reveals significant variability in performance based on task type and model architecture:

• Task Dependency: AI tools perform best on discussion tasks (e.g., explaining model behavior, suggesting steps), where LLMs excel at reorganizing provided text. Performance drops significantly on quantitative error fixing and causal reasoning tasks requiring deep domain logic.
• Qualitative vs. Quantitative: Tools generally pass basic competency tests for qualitative tasks (translation, iteration, conformance) but struggle with causal reasoning grounded in real-world examples. Quantitative tools show limited competence, particularly in identifying and fixing formulation errors.
• No Dominant LLM: No single LLM dominates across all engine types or tasks. High-performing models in one category often underperform in another.
• Speed vs. Accuracy Trade-offs: The most advanced or "highest-reasoning" LLMs do not consistently outperform smaller, faster models. In some cases, more complex models "overthink," producing verbose outputs with higher latency but no commensurate accuracy gains. Discussion tasks are fastest (25s), while quantitative tasks are slowest (60s).
• Specific Findings:
  • Qualitative: High scores in translation and conformance; low scores in causal reasoning (e.g., some models scored 0% on causal reasoning).
  • Quantitative: Moderate scores in translation and iteration; very low scores in suggested model fixes (ranging from 0% to 42.9%).
  • Discussion: High overall scores (up to 94.1%), particularly in suggesting model building steps and fixes.

Significance
The paper posits that the BEAMS Initiative provides a necessary shift from natural language processing assessments to domain-specific tests that reflect the demands of real modeling work. By establishing transparent, reproducible, and openly governed benchmarks, the initiative:

1. Incentivizes Responsible Development: Creates a mechanism for the modeling community to drive LLM builders toward tools that are interpretable, ethical, and aligned with human expertise.
2. Validates Human-AI Collaboration: Demonstrates that AI is most effective as a cognitive aid that accelerates routine steps (e.g., drafting links) while keeping the human modeler "in the loop" for validation and ethical scrutiny.
3. Reveals Task-Specific Nuances: Highlights that "better" AI is not a monolithic concept; performance is highly dependent on task framing, suggesting a need for flexible strategies like per-task LLM routing rather than relying on a single "best" model.
4. Foundation for Future Work: While current evaluations align with most design principles, the initiative acknowledges ongoing efforts to address bias and incorporate alternative perspectives, positioning the framework as a cumulative, community-driven standard for the future of AI in simulation.