Technical Summary: Specialists or Generalists? Multi-Agent and Single-Agent LLMs for Essay Grading

Problem Statement

Automated Essay Scoring (AES) systems increasingly rely on Large Language Models (LLMs), yet the literature lacks clarity on how architectural choices—specifically the trade-off between single-agent (monolithic) and multi-agent (hierarchical) designs—impact performance across different essay quality levels. While proponents argue that decomposing tasks into specialized roles enhances interpretability and agreement, empirical evidence remains mixed, with some studies suggesting single-agent strategies with few-shot prompting outperform complex multi-agent workflows. This study addresses the gap in understanding how these architectures interact with essay quality granularly, questioning whether the increased computational cost of multi-agent systems is justified across the entire score distribution.

Methodology

The study conducted a rigorous comparison of Single-Agent and Multi-Agent architectures using the ASAP 2.0 corpus, a dataset of 24,278 source-based argumentative essays from U.S. secondary students (grades 6–10). The researchers extracted a stratified test set of 450 essays and a separate calibration set of 12 essays (2 examples per score level, 1–6).

Experimental Setup

All experiments utilized the GPT-5.1 model via the OpenAI API under two prompting conditions:

1. Zero-Shot: The model received only the official rubric.
2. Few-Shot: The model received the rubric plus the 12 calibration examples.

Architectural Designs

• Single-Agent (Monolithic): The model acted as a sole expert, receiving the full rubric and generating a holistic score (1–6) and rationale.
• Multi-Agent (Hierarchical): This system decomposed grading into three Specialist Agents and one Chairman Agent:
  • Specialists: Content (ideas/evidence), Structure (organization/coherence), and Language (mechanics/vocabulary). Each was instructed via negative constraints to ignore dimensions outside their domain.
  • Chairman Agent (Adjudicator): Received reports from specialists and applied rubric-aligned logic-based heuristics rather than simple averaging. Key rules included:
    • Veto Rule: If any specialist assigned a score of 1, the final score is 1.
    • Capping Logic: If any specialist assigned a 2, the final score is capped at 2 or 3.

Evaluation Metrics

Performance was measured against human ground truth using Quadratic Weighted Kappa (QWK) to penalize large discrepancies and Exact Match Accuracy (percentage of predictions matching human scores exactly).

Key Results

Overall Performance

The Multi-Agent system with Few-Shot prompting achieved the highest overall performance (QWK: 0.7453; Exact Match: 57.11%), outperforming the Single-Agent Few-Shot baseline (QWK: 0.7165; Exact Match: 54.00%) by a marginal margin.

Impact of Few-Shot Calibration

Few-shot calibration emerged as the dominant factor for both architectures. Providing just two examples per score level improved QWK by approximately 26% for both systems:

• Single-Agent: QWK increased from 0.5664 (Zero-Shot) to 0.7165.
• Multi-Agent: QWK increased from 0.5917 (Zero-Shot) to 0.7453.

Performance by Score Level

Granular analysis revealed distinct strengths and weaknesses:

• Low-Quality Essays (Scores 1–2): The Multi-Agent system significantly outperformed the Single-Agent system. For Score 1, Multi-Agent Few-Shot achieved 73.3% accuracy versus 46.7% for Single-Agent. This is attributed to the Chairman Agent's veto and capping rules, which effectively penalize critical deficiencies in any single dimension.
• Mid-Range Essays (Scores 3–4): The Single-Agent system demonstrated superior or competitive performance. For Score 4, Single-Agent Few-Shot achieved 61.2% accuracy compared to 57.3% for Multi-Agent. The authors suggest the holistic view of the single agent allows for implicit compensatory scoring (balancing strengths and weaknesses) that specialist agents, constrained by narrow domains, cannot replicate.
• High-Quality Essays (Scores 5–6): Both architectures struggled significantly. For Score 5, the best configurations achieved only 30.8% accuracy. For Score 6 (only 3 essays in the test set), both few-shot configurations correctly identified only 1 essay (33.3%). The Multi-Agent system showed a conservative bias, often under-predicting high scores due to its capping logic.

Key Contributions

The paper makes three primary contributions to the field of AES and multi-agent LLM systems:

1. Empirical Comparison of Architectures: It provides a granular analysis showing that architectural choice fundamentally shapes performance patterns in ways that aggregate metrics (like overall QWK) obscure. It demonstrates that neither architecture universally dominates; rather, they exhibit complementary strengths based on essay quality.
2. Novel Multi-Agent Design: The study introduces a multi-agent framework combining domain decomposition with logic-based aggregation rules (veto and capping) aligned with official rubrics, moving beyond simple voting or averaging mechanisms.
3. Deployment Guidance: It offers practical insights indicating that few-shot calibration is essential for both architectures, that multi-agent systems incur 4x computational costs (four API calls per essay) for only marginal overall gains, and that both systems struggle with exceptional essays.

Significance and Implications

The authors conclude that architectural selection should be driven by specific deployment priorities rather than overall metrics alone:

• Multi-Agent Systems: Best suited for diagnostic screening and identifying at-risk students (low-quality essays) where detecting critical deficiencies is paramount. They also offer enhanced interpretability through dimension-specific feedback. However, they come with higher costs and a conservative bias against high scores.
• Single-Agent Systems: Provide a cost-effective solution for general assessment, particularly for mid-range essays where nuanced trade-offs exist. They are more efficient but lack the diagnostic depth and robustness in flagging low-quality work.

The study emphasizes that few-shot calibration is a critical implementation requirement, noting that the substantial performance gains from a small number of examples make this a necessary investment for system effectiveness. Finally, the authors acknowledge limitations regarding dataset generalizability and the need for future research into bias mitigation and the evaluation of the interpretability of the multi-agent feedback.