Technical Summary: Large Language Models as Supervised Extraction Assistants in Agent-Based Modelling

1. The Problem: Documentation Standards and the Adoption Barrier

Agent-Based Modelling (ABM) relies on transparent reporting of methodological choices and data usage to ensure credibility and reproducibility. While standards exist to facilitate this—most notably the ODD (Overview, Design concepts, and Details) protocol for model specification and the RAT-RS (Rigour And Transparency Reporting Standard) for data use—adoption remains limited. The primary barrier is the significant effort required to produce documentation, which is often viewed as supplementary rather than core content. This creates a "Tragedy of the Commons" scenario: while widespread adoption would enhance the collective rigour and credibility of the field, individual researchers are deterred by the time cost and lack of immediate benefit.

The paper posits that Large Language Models (LLMs) offer a novel strategy to lower this barrier. By acting as "supervised extraction assistants," LLMs could perform the initial, labor-intensive work of scanning publications, identifying relevant data, and mapping information to structured protocols, thereby shifting the human researcher's role from producer to constructive critic.

2. Methodology: A Feasibility Study on Supervised Extraction

This paper presents a feasibility study rather than a statistically generalisable claim. The study evaluates the extent to which LLMs can support the extraction and augmentation of data-use reports from published ABM papers using the RAT-RS standard.

2.1 Experimental Design

• Target Standard: The study focuses on the RAT-RS, chosen for its granular, question-based structure (originally 39 questions) which makes it suitable for interaction with LLMs. The authors extended the standard with a sixth "Question Suite" (QS) containing five meta-evaluative questions, resulting in a total of 44 questions.
• Case Study: The feasibility test used a single published paper: Siebers and Aickelin (2011), "A First Approach on Modelling Staff Proactiveness in Retail Simulation Models." This paper was selected because a manually completed RAT-RS report already existed, providing a human-authored ground truth for comparison.
• Models Tested: Four state-of-the-art LLMs were evaluated using their internal reasoning modes: Claude Sonnet 4.6, DeepSeek-V3, ChatGPT-5.1, and Gemini Flash 3.
• Prompt Engineering: A modular, evidence-based prompt was designed incorporating:
  • A specific expert persona.
  • Sequential step decomposition (Chain-of-Thought).
  • Explicit grounding rules to suppress hallucinations.
  • A four-category epistemic taxonomy for signalling evidential certainty.
  • A constrained output format (CSV, 100-word limit per response).
  • Few-shot learning via an anonymised worked example.

2.2 Evaluation Metrics

The study employed two complementary dimensions of analysis:

1. Cross-Comparison: Comparing human responses against LLM responses (Human vs. LLM) and comparing responses across different models (LLM vs. LLM).
2. Internal Consistency: Running the same prompt eight times on the same paper using DeepSeek to assess intra-model stability.
3. Semantic Similarity: Pairwise cosine similarity scores were calculated to measure topical overlap between responses.
4. Question Type Taxonomy: Questions were classified into five types to analyze performance variations:
   • Factual: Objectively verifiable data (e.g., Q1.1).
   • Descriptive: Characterising features using "what/which" (e.g., Q1.2).
   • Explanatory: Requiring causal reasoning or justification ("why") (e.g., Q1.6).
   • Binary: Yes/No with conditional elaboration.
   • Evaluative: Meta-level appraisal of the paper.

3. Key Results and Findings

3.1 Semantic Alignment and Divergence

• Human vs. LLM: The average cosine similarity between human and LLM responses was low (0.497). This divergence was largely attributed to stylistic differences: human responses were shorter, informal, and analogy-rich, while LLM responses were longer, formal, and policy-citing. Crucially, this low similarity did not necessarily indicate informational disagreement; for factual and descriptive questions, LLMs often matched or exceeded human detail.
• LLM vs. LLM: Inter-model similarity was moderate (0.586), indicating that different models share broadly similar training conventions and response formats.
• Intra-Model Consistency: Repeated runs of the same model showed moderate-to-high consistency (0.664 average), suggesting the extraction process is stable for specific question types.

3.2 The "Justification Penalty"

A critical finding is the significant drop in consistency and alignment when moving from descriptive to explanatory questions.

• Descriptive Questions: Showed high consistency (mean ~0.750) and alignment. The answers are grounded in specific text sections.
• Explanatory Questions: Showed significantly lower consistency (mean ~0.551) and alignment. These questions require the model to infer causality and construct narratives, leading to higher variability.
• The Penalty: The study identifies a "justification penalty" of approximately 0.199 points, representing the drop in performance when a question requires reasoning rather than simple extraction.

3.3 Qualitative Observations

• Hallucinations: Outright fabrications were rare. When LLMs inferred missing information, they generally framed it as an inference rather than a confident statement, adhering to prompt instructions.
• Nuance: LLMs sometimes omitted nuances present in human answers or added unnecessary conditional logic.
• Utility: LLMs proved capable of synthesising information from multiple paper sections, often providing more elaborate answers than humans for descriptive tasks.

4. Key Contributions and Practical Heuristics

The paper does not claim LLMs can replace human judgement but argues they can make standard adoption practical at scale through Supervised Extraction. The authors propose three conditions for reliable use:

1. Human-in-the-Loop Triage:

   • High Confidence: Descriptive and factual questions (e.g., Calibration and Administrative blocks) can be treated as high-confidence outputs requiring minimal review.
   • High Review Priority: Explanatory and evaluative questions (e.g., Data Existence and Reflective Appraisal blocks) require thorough human review.
   • Diagnostic Signal: High variability in repeated LLM runs for a specific question signals that the underlying source text is ambiguous or missing data, guiding the human reviewer to focus on those gaps.

2. Transparency and Disclosure:

   • Reports generated via supervised extraction must explicitly disclose the LLM used, the prompt version, the number of runs, and the nature of human oversight.

3. Community-Driven Prompt Engineering:

   • Prompt quality is critical. The authors advocate for a shared repository of versioned prompts and validation cases to improve robustness across different ABM traditions and data types.

5. Significance and Limitations

Significance:
The paper demonstrates that LLMs can serve as effective drafting and extraction assistants, potentially lowering the barrier to adopting rigorous documentation standards like RAT-RS. By automating the "hard work" of reading and initial mapping, LLMs allow researchers to focus on verification and interpretation. This shifts the documentation process from a prohibitive cost to a manageable workflow, potentially transforming aspirational standards into attainable practices.

Limitations:

• Scope: The study is a feasibility test based on a single paper and does not claim statistical generalisability across the entire ABM literature.
• Metrics: Cosine similarity is used as a coarse proxy for topical overlap, not a measure of correctness or argumentative equivalence.
• Reproducibility: The use of commercial, non-deterministic LLMs necessitates multi-run sampling and strict metadata reporting (model version, settings) to ensure reproducibility.
• Content vs. Form: LLMs cannot create rich documentation from sparse papers; they can only extract what is present (or infer it with caveats).

6. Call to Action

The authors call on the ABM community to:

1. Adopt the provided prompt templates and tools for systematic extraction.
2. Contribute validation cases by applying the workflow to their own papers and submitting results to a shared repository to build a field-validated dataset.
3. Update standards (e.g., ODD, RAT-RS) to formally acknowledge and define protocols for AI-assisted reporting, including guidelines on which question types can be automated and which require mandatory human review.

In conclusion, the paper argues that while LLMs do not solve all documentation challenges, they represent a pivotal step in making rigorous reporting feasible for researchers under time constraints, bridging the gap between well-discussed standards and practical adoption.