Developing a Multi-Agent System to Generate Next Generation Science Assessments with Evidence-Centered Design

1. Problem Statement

The paper addresses the critical bottleneck in modern science education: the difficulty of scaling Next Generation Science Standards (NGSS) assessments.

• The Challenge: NGSS requires "knowledge-in-use" assessments where students apply Disciplinary Core Ideas (DCI), Science and Engineering Practices (SEP), and Crosscutting Concepts (CCC) to solve problems. Designing these requires Evidence-Centered Design (ECD), a rigorous framework linking learning goals to observable evidence.
• The Bottleneck: Traditional ECD implementation is labor-intensive, requiring diverse human expertise (content, measurement, and technical specialists) to manually define domain models, evidence models, and task models.
• The Gap in AI: While Automated Item Generation (AIG) using Large Language Models (LLMs) exists, current approaches often lack a solid measurement foundation. They frequently fail to ensure validity, reliability, or alignment with learning goals, and they struggle with multimodal coherence (integrating text, images, and data).

2. Methodology: The ECD-based Multi-Agent System (MAS)

The authors propose a Multi-Agent System (MAS) that operationalizes the ECD framework by decomposing the assessment design process into specialized roles, each powered by an LLM (GPT-4o-mini).

System Architecture

The system follows a unidirectional, inheritance-based information flow where the output of one agent serves as the structured input (JSON) for the next. It consists of five specialized agents:

1. Domain Modeling Agent: Identifies the Performance Expectation (PE) and extracts the specific DCI, SEP, and CCC components to anchor the design.
2. Evidence Modeling Agent: Expands the PE into explicit Knowledge, Skills, and Abilities (KSAs), defining observable behaviors, partial understandings, and misconceptions.
3. Scenario Design Agent: Generates daily-life scenarios that elicit the specific evidence behaviors defined by the previous agent, ensuring context and task conditions are appropriate.
4. Item Generation Agent: Transforms the scenario into a full assessment task, including prompts, expected responses, scoring rubrics, and fairness considerations.
5. Quality Evaluation Agent: An independent agent that performs a pass/fail judgment against predefined NGSS criteria (alignment, cognitive demand, clarity). If the item fails, it triggers a regeneration loop (max 3 retries).

Implementation Details

• Multimodal Generation: After finalizing the text, the system uses Gemini Imagen 4.0 to generate visual representations (images, graphs) based on structured prompts to ensure visual coherence.
• Constraints: The system enforces strict constraints to prevent "construct drift," ensuring KSAs remain strictly aligned with NGSS dimensions and that evidence is instantiated as observable data (tables/graphs).

Evaluation Design

• Dataset: A comparative study of 60 items (30 human-developed NGSA items vs. 30 MAS-generated items) across Life Science and Physical Science domains.
• Rubric: A multidimensional rubric covering:
  • NGSS 3D Alignment (PE, SEP, DCI, CCC).
  • Cognitive Demand (Depth of Knowledge levels).
  • Student Engagement (Relevance/Interest, Inclusivity).
  • Language Quality (Clarity, Conciseness, Appropriateness).
  • Multimodal Design (Coherence, Signaling, Spatial Contiguity).
• Raters: Two science education experts scored the items, with inter-rater reliability established (AC2 = 0.90 for most components).

3. Key Results

The study found that MAS-generated items are comparable in overall quality to human-developed items regarding standards alignment and cognitive demand, but exhibit distinct strengths and weaknesses.

Dimension	MAS Performance	Human Performance	Key Finding
NGSS 3D Alignment	100% Satisfactory	100% Satisfactory	Both groups successfully targeted PEs and integrated DCI/SEP/CCC.
Cognitive Demand	Mostly DOK 3-4	Mostly DOK 3-4	Both groups successfully generated high-order thinking tasks.
Evidence Collectability	Weaker (Higher "Not Evident" rate)	Stronger	MAS items often failed to elicit the specific evidence required for scoring due to prompt mismatches.
Inclusivity	Stronger (100% Evident)	Mixed	MAS used more neutral, non-deficit language; human items occasionally relied on specific cultural contexts.
Linguistic Clarity	Weaker (High "Partially Evident")	Strong	MAS items suffered from redundancy and grammatical errors in image captions/annotations.
Multimodal Design	Significantly Weaker	Strong	MAS visuals often had internal inconsistencies (e.g., mismatched axes/units) or poor readability.
Student Interest	Mixed	Mixed	Both groups struggled to align items with high student interest.

Qualitative Themes:

• Prompt Completeness: MAS often generated incomplete prompts (e.g., "answer the following questions" without listing them).
• Context Trade-off: MAS prioritized inclusivity through generic scientific terms, while human items used concrete everyday examples (which sometimes lacked universal accessibility).
• Visual Degradation: MAS-generated images frequently contained typographical errors, overlapping text, or mismatched data, rendering them unusable for student reasoning.

4. Key Contributions

1. Operationalizing ECD in AI: The study successfully translates the theoretical ECD framework (Claims-Evidence-Tasks) into a functional, automated Multi-Agent System, moving beyond simple "post-hoc" alignment checks to structural constraints during generation.
2. Scalable Assessment Design: It demonstrates that MAS can automate the labor-intensive steps of ECD (domain modeling, evidence specification), offering a pathway to scale NGSS-aligned assessments.
3. Human-in-the-Loop Necessity: The results provide empirical evidence that while AI can handle the structure and alignment of assessments, human expertise remains essential for refining linguistic clarity, ensuring multimodal coherence, and verifying evidence collectability.
4. Multimodal Limitations: The paper highlights a critical current limitation in generative AI: the inability to consistently generate coherent, interpretable visual data representations (graphs, tables) that align perfectly with text prompts.

5. Significance and Future Directions

• Practical Impact: The ECD-based MAS serves as a powerful support tool for teachers and assessment developers, capable of generating drafts, checking alignment, and accelerating the initial design phase. It reduces the time commitment for ECD activities but does not replace the need for human review.
• Theoretical Shift: It bridges the gap between educational measurement theory (ECD) and generative AI, proving that theoretical frameworks can be encoded into agent workflows to ensure validity.
• Future Work: The authors suggest future research should focus on:
  • Defining optimal points for human intervention in the AI workflow.
  • Improving the consistency of multimodal generation (text-image alignment).
  • Testing the framework in real-world classroom settings to evaluate usability and student learning outcomes.

Conclusion: Integrating ECD into a Multi-Agent System is a viable strategy for scalable, standards-aligned assessment design. However, the current technology acts as a co-pilot rather than an autonomous pilot, requiring human oversight to ensure the final assessment is clear, coherent, and valid.