Quality-Conditioned Agreement in Automated Short Answer Scoring: Mid-Range Degradation and the Impact of Task-Specific Adaptation

Technical Summary: Quality-Conditioned Agreement in Automated Short Answer Scoring

Problem Statement
Automated Short Answer Scoring (ASAS) is transitioning from discriminative, fine-tuned models to Large Language Models (LLMs) utilized in few-shot settings. While LLMs offer broad world knowledge and ease of deployment, their reliance on limited task-specific data raises concerns regarding alignment with expert grading, particularly for complex tasks requiring nuanced interpretation. A critical, underexplored issue is how these models perform on "mid-range" responses—answers that are partially correct and require distinguishing between correct and incorrect elements. Existing literature suggests that automated scoring accuracy may vary systematically across response quality, potentially leading to inequitable evaluations for students with developing understanding. The central research question investigates the association between response quality and inter-rater agreement for human-human versus human-AI pairings across different levels of task-specific adaptation (varying few-shot examples vs. fine-tuning).

Methodology
The study evaluated scoring agreement on two open-ended biology items concerning cellular respiration (specifically the effects of smoking and anemia on exercise). The dataset comprised approximately 800 high school student responses.

• Ground Truth: Responses were scored by a biology education expert (Rater1) using an analytic rubric decomposing explanations into 10 binary categories. This scoring served as the ground truth.
• Models Compared:
  1. Human Expert (H2): A second biology education expert.
  2. Fine-Tuned Encoder: A BERT-based discriminative model (DictaBERT) fine-tuned on 669 training responses.
  3. Few-Shot LLMs: Three generative decoder-based models (GPT-5.2, GPT-4o, and Claude Opus 4.5) prompted with varying numbers of examples (2, 4, 6, 8, and 10-shot).
• Evaluation Metric: Scoring differences were measured using Manhattan Distance (L1 distance) between the binary vector of the ground truth and the vector produced by the rater/AI. This metric counts the number of rubric categories scored differently.
• Experimental Design: The study analyzed agreement across the full spectrum of response quality (0 to 10 correct categories), specifically examining performance at the extremes (fully correct/incorrect) versus the mid-range (partially correct).

Key Results

1. Mid-Range Degradation: All AI models exhibited a U-shaped performance pattern (or inverted U-shaped agreement). While models performed well on fully correct (scores 9–10) and fully incorrect (scores 0–1) responses, they showed substantial degradation in the mid-range (scores 4–7).
2. Human vs. AI Consistency: The human expert (H2) maintained high and stable agreement with the ground truth across the entire quality spectrum, showing no significant mid-range degradation. In contrast, AI graders were less aligned with the expert specifically on mid-range responses.
3. Impact of Task-Specific Adaptation: The severity of mid-range degradation was conditioned on the amount of task-specific data:
   • Few-Shot LLMs: Models with fewer examples (e.g., 2-shot) showed the most severe degradation in the mid-range. Increasing the number of examples (up to 10-shot) reduced but did not eliminate this gap.
   • Fine-Tuned Models: The fine-tuned BERT-based model performed best among the AI systems, significantly reducing the mid-range error compared to few-shot LLMs, though it still did not match the stability of the human expert.
4. Model Variability: While GPT-4o and GPT-5.2 showed consistent patterns, the Claude Opus 4.5 model exhibited erratic behavior, performing best on one item but worst on the other, suggesting instability in few-shot prompting for this specific architecture.

Key Contributions

• Identification of Quality-Conditioned Fairness: The paper introduces the concept of "quality-conditioned fairness," defining it as the requirement for ASAS systems to maintain consistent accuracy across the entire range of response quality.
• Characterization of Mid-Range Bias: The study empirically demonstrates that the primary source of disagreement between AI and human experts is not at the extremes of the score distribution, but specifically in the mid-range where students demonstrate partial mastery.
• Adaptation vs. Performance: The work clarifies that while LLMs can handle extreme cases with minimal data, accurate evaluation of partial knowledge requires significant task-specific adaptation (fine-tuning or extensive few-shot examples).

Significance and Claims
The authors claim that these findings highlight a critical limitation in the current paradigm of deploying few-shot LLMs for ASAS without sufficient task-specific data. The "mid-range degradation" poses a specific fairness risk: students with developing knowledge states are the most likely to receive inaccurate scores, which could lead to inappropriate learning support or feedback.

The paper suggests a practical hybrid approach: automated systems could reliably handle fully correct and fully incorrect responses (even with few-shot prompting), while mid-range responses should be routed to human raters until sufficient task-specific data is accumulated for fine-tuning. By identifying this specific bias pattern, the authors argue that the field can move from treating scoring errors as "unknown biases" to "known biases," a necessary step for improving fairness in educational AI. The study does not claim that LLMs are inherently superior or inferior to fine-tuned models in all contexts, but rather that their performance is highly dependent on the quality of the response being scored and the degree of adaptation applied.