MADRAG: Multi-Agent Debate with Retrieval-Augmented Generation for Training-Free Analytic Essay Scoring

Technical Summary: MADRAG for Training-Free Analytic Essay Scoring

Problem Statement
Automated Essay Scoring (AES) faces a persistent trade-off between flexibility and reliability. Traditional supervised systems achieve high reliability but require task-specific training data for every new prompt and rubric, limiting their adaptability in dynamic classroom environments. Conversely, training-free Large Language Model (LLM) approaches offer flexibility but suffer from poor calibration, sensitivity to prompt design, and systematic biases—most notably a "middle-score bias" where models regress toward the center of the scoring scale, failing to distinguish clearly between exceptional and weak inputs. Furthermore, standard LLM-as-judge approaches often lack the structured reasoning required for reliable analytic trait scoring (evaluating dimensions like ideas, organization, and conventions separately), leading to inconsistent feedback.

Methodology: MADRAG Framework
The authors propose MADRAG (Multi-Agent Debate with Retrieval-Augmented Generation), a fully training-free framework designed to approximate human-level reliability without supervised fine-tuning. The system decomposes the scoring process into two complementary mechanisms:

1. Multi-Agent Debate (MAD):

   • Advocate: An agent that analyzes the essay and rubric to generate an argument focusing exclusively on the essay's strengths.
   • Skeptic: An agent that reviews the Advocate's argument and the rubric to generate a rebuttal focusing exclusively on weaknesses.
   • Judge: An impartial agent that synthesizes the debate transcript, the rubric, and confidence scores (derived from the log-probability of the first token generated by the Advocate and Skeptic) to produce a final integer score.
   • This structure forces explicit deliberation, preventing the model from relying on a single, potentially biased intuition.

2. Retrieval-Augmented Generation (RAG):

   • Before scoring, the Judge retrieves rubric-aligned exemplar essays from a vector database.
   • The retrieval is trait-specific: for a given trait (e.g., "Organization"), the system retrieves one exemplar for each possible score value within the valid range.
   • These exemplars serve as a "few-shot" calibration reference, grounding the Judge's decision in concrete examples of what constitutes a score of 1, 2, 3, etc., rather than relying on abstract internal representations.

The workflow is orchestrated by a Supervisor agent that decomposes the rubric, manages the retrieval of few-shot examples for each trait, and coordinates the debate between the Advocate, Skeptic, and Judge.

Key Contributions

• Training-Free High-Performance Scoring: MADRAG demonstrates that a system without parameter updates or supervised fine-tuning can achieve trait-level scoring reliability comparable to state-of-the-art supervised AES models.
• Calibration via External Memory: The paper identifies that retrieval of score-calibrated exemplars is the primary driver for mitigating middle-score bias, allowing the model to distinguish extreme scores effectively.
• Structured Reasoning for Complex Traits: The multi-agent debate mechanism is shown to improve reasoning on higher-level, discourse-oriented traits (e.g., Ideas, Organization) by encouraging the synthesis of conflicting viewpoints.
• Ablation of Components: The study isolates the specific contributions of decomposition, debate, and retrieval, showing that while retrieval drives calibration, debate refines reasoning on complex traits, though it may introduce noise for surface-level error counting.

Experimental Results
Evaluated on the ASAP dataset (Essay Sets 7 and 8), MADRAG was compared against supervised baselines (e.g., ProTACT, FeatEng-RF) and training-free LLM baselines (e.g., ZS-LLM, CSR-J).

• Overall Performance: MADRAG significantly outperforms all prior training-free approaches. On the "Ideas" trait, MADRAG (using GPT-5) achieved a Quadratic Weighted Kappa (QWK) of 0.75 in Set 7, surpassing the supervised ProTACT model (0.50) and exceeding Human-Human agreement (0.69).
• Trait Dependency: Gains were most pronounced on discourse-oriented traits (Ideas, Organization), where MADRAG often matched or exceeded supervised systems. Performance on surface-level traits (e.g., Conventions, Word Choice) showed more modest improvements and greater variance, suggesting that debate structures optimized for holistic argumentation are less effective for counting specific mechanical errors.
• Bias Mitigation: In an analysis of essays with extreme human scores (minimum or maximum), MADRAG achieved the highest agreement (Agree@1) and lowest Mean Absolute Error (MAE). Ablation studies confirmed that retrieval was the dominant factor in reducing middle-score bias (MARAG outperformed MAD), while debate provided additional refinement on top of retrieval.
• Error Analysis: Qualitative analysis of failure modes revealed that the most common errors were "debate framing capture" (32.7%) and "anonymization distortion" (26.9%), where the model misinterpreted placeholders (e.g., @PERSON) as errors. Most errors were "partially grounded," meaning the rationale sounded rubric-consistent but lacked decisive text evidence.

Significance and Claims
The paper claims that MADRAG illustrates how hybrid LLM frameworks can combine the flexibility of prompt-based scoring with the reliability of supervised systems. By integrating adversarial reasoning with external memory (exemplar retrieval), the framework produces calibrated, interpretable, trait-level assessments without the need for task-specific training data.

The authors emphasize that while MADRAG does not require parameter updates, it is not "data-free," as it relies on a bank of manually scored essays for retrieval. However, the data requirement is significantly lower than that of supervised models. The work highlights the critical role of structured interaction (debate) and external memory (retrieval) in overcoming the inherent calibration and consistency limitations of single-agent LLM judges.

Limitations
The authors acknowledge that their results are limited to narrative essays from middle and high school settings (ASAP Sets 7 and 8) and may not fully transfer to other genres, languages, or rubric structures. Additionally, the system is computationally more demanding than single-agent prompting due to multiple LLM calls and retrieval steps per trait. Finally, the presence of anonymization placeholders in the dataset introduced specific biases for surface-level traits, a challenge that may persist in real-world deployments.