Evaluating Austrian A-Level German Essays with Large Language Models for Automated Essay Scoring

This paper evaluates the performance of four state-of-the-art open-weight Large Language Models on Austrian A-level German essay grading using rubric-based prompts, finding that while they can apply standardized criteria, their low agreement rates with human experts (maximum 40.6% on sub-dimensions and 32.8% on final grades) render them currently unsuitable for real-world automated scoring.

The Gist

Imagine you are a teacher who has to grade hundreds of essays every week. It's exhausting, time-consuming, and sometimes, you might be a little too harsh on a Tuesday morning or too lenient on a Friday afternoon. This is the problem Automated Essay Scoring (AES) tries to solve: using computers to do the grading so teachers can focus on teaching.

For a long time, computers were like rigid robots that could only count how many words were in an essay or check for spelling mistakes. They couldn't really "understand" the writing. But recently, we have these super-smart AI brains called Large Language Models (LLMs) that can read, write, and reason almost like humans.

This paper is like a report card for a specific experiment: Can these new AI brains grade Austrian high school German essays as well as a human teacher?

Here is the breakdown of what they did, using some everyday analogies:

1. The Setup: The "Exam Hall"

The researchers gathered 101 real student essays from Austrian high school exams. These weren't just random paragraphs; they were specific types of writing like opinion pieces, letters to the editor, and literary analyses.

Think of the grading system like a recipe. In Austria, there is a strict "recipe" (called a Rubric) that teachers must follow. It breaks the essay down into four ingredients:

• Content (Did they say the right things?)
• Structure (Is it organized well?)
• Style (Does it sound good?)
• Language (Is the grammar correct?)

The goal was to see if an AI could follow this recipe perfectly.

2. The Contestants: The AI Models

They tested four different AI models (DeepSeek, Qwen, Mixtral, and Llama). Imagine these as four different student interns hired to help grade the papers.

• Mixtral was like an intern who got bored and just gave everyone a "C" (Grade 3) without reading.
• DeepSeek was an intern who was too strict, gave weird feedback, and sometimes even spoke Chinese instead of German!
• Qwen was okay but still a bit too harsh.
• Llama3.3 was the clear winner. It was the most reliable, understood the instructions best, and gave the most human-like feedback.

3. The Training: How did they teach the AI?

The researchers tried three different ways to help the AI understand what a "good" grade looks like:

• The "Zero-Shot" Approach (The Blind Guess): They just handed the AI the essay and the rules.
  • Result: The AI was confused. It didn't know what a "Grade 1" (perfect) actually looked like compared to a "Grade 5" (fail).
• The "RAG" Approach (The Library): They gave the AI a "cheat sheet" of other essays (some good, some bad) to look at while grading.
  • Result: Better, but the AI still struggled to distinguish between the very best and the very worst essays. It was like giving a student a textbook but not telling them which page to read.
• The "Few-Shot" Approach (The Mentorship): This was the best method. They showed the AI a few examples: "Here is a perfect essay (Grade 1), here is a messy one (Grade 5), and here is a middle-of-the-road one (Grade 3)." Then they asked the AI to grade a new essay based on those examples.
  • Result: This worked the best. It was like showing the intern a portfolio of past work so they could get a feel for the standards.

4. The Results: Did they pass the test?

Here is the bad news: The AI is not ready to replace human teachers yet.

• The Agreement Rate: When the AI and the human teacher graded the same essay, they only agreed on the final grade about 33% of the time.
• The Sub-scores: Even when looking at just the "Content" or "Grammar," the AI only agreed with the human about 40% of the time.

Think of it like this: If a human teacher gives an essay an "A," the AI might give it a "C" or a "B." It's not totally random, but it's not accurate enough to be trusted with a real student's future.

5. Why did it fail?

The paper points out a few reasons why the AI struggled:

• The "Black Box" Problem: The AI sometimes guessed the middle ground (Grade 3) because it was scared to be too extreme.
• The "Handwriting" Hurdle: Many student essays were handwritten. The computer had to scan them and turn them into text (OCR). This process introduced errors, like turning a messy "s" into an "f," which confused the AI.
• The "One Teacher" Bias: The dataset only had grades from one human teacher. If that teacher was a bit quirky or strict, the AI learned that specific style, not the "universal" truth of grading.

The Bottom Line

This paper is a reality check. While AI has made huge leaps, it's currently more like a helpful teaching assistant than a head teacher.

It can read an essay, check for grammar, and give a rough idea of the quality. But it can't yet understand the nuance of a student's unique voice or the subtle differences between a "good" and a "great" argument in the way a human can.

The Future: The authors suggest that in the future, AI won't replace teachers. Instead, it will be a tool that does the boring work (checking spelling, counting words, suggesting structure) so teachers can spend their time giving the real human feedback that students need. But for now, we still need humans in the loop to make sure the grades are fair.

Technical Summary

Here is a detailed technical summary of the paper "Evaluating Austrian A-Level German Essays with Large Language Models for Automated Essay Scoring."

1. Problem Statement

The paper addresses the challenge of Automated Essay Scoring (AES) for Austrian A-level (Matura) German examinations.

• Context: Austrian A-level exams involve seven distinct text types (e.g., Discussion Essay, Commentary, Literary Interpretation), each requiring specific grading rubrics.
• Pain Points: Human grading is time-consuming (teachers spend ~13.5% of work time on grading) and prone to subjective bias.
• Gap: While Large Language Models (LLMs) have shown promise in AES, there is a lack of rigorous evaluation regarding their ability to handle multiple text types simultaneously using standardized, rubric-based criteria in a non-English (German) context. Existing benchmarks often lack the complexity of real-world national exams.

2. Methodology

The study evaluates the performance of state-of-the-art open-weight LLMs using a dataset of 101 anonymized student exams covering three text types: Commentary, Letter to the Editor, and Literary Interpretation.

2.1 Models Evaluated

Four LLMs were initially tested, though only one was used for deep analysis due to performance disparities:

• LLaMA 3.3 70b (Primary model)
• DeepSeek-R1 32b
• Qwen3 30b
• Mixtral 8x7b

2.2 Experimental Design

The researchers compared several prompting and context strategies to determine the most effective approach for rubric-based grading:

1. Baseline (Zero-Shot): Providing the rubric and text without examples.
2. Retrieval Augmented Generation (RAG):
   • Best-Average-Worst: Fixed context of three reference texts (top, middle, bottom scores).
   • Most-Similar-Matches: Vector-based retrieval of similar texts using german-roberta-sentence-transformer-v2.
   • Range-of-Examples: Retrieving sets of 5–10 texts to cover the full grading spectrum (1–5).
3. Few-Shot Prompting: Iterative in-context learning where the model grades texts and receives feedback/correct answers before grading the next.
   • Tested various sequences (Best-to-Worst, Inverted, Mixed).
4. Chain-of-Thought (CoT): Adding reasoning steps to the few-shot prompts to improve logical consistency.

2.3 Evaluation Metrics

Performance was measured against human expert grades using:

• Quadratic Weighted Kappa (QWK): Measures inter-rater agreement, accounting for the severity of errors.
• Mean Absolute Error (MAE): Average deviation from the human grade.
• Pearson Correlation Coefficient (PCC): Linear relationship between predicted and actual grades.
• Accuracy: Percentage of exact grade matches.

3. Key Contributions

• Comprehensive Rubric Evaluation: The study is one of the first to test LLMs on the specific, multi-dimensional grading rubrics of Austrian A-levels, which require scoring four distinct sections (Content, Structure, Style/Expression, Language Conventions) across different text genres.
• RAG vs. Few-Shot Comparison: It provides empirical data comparing retrieval-based context (RAG) against iterative few-shot learning for AES, highlighting that few-shot prompting generally outperforms RAG for final grade accuracy, while RAG offers better stability in sub-dimensions.
• Model Selection Insights: The paper documents significant failure modes in smaller or non-German-optimized models (e.g., Mixtral defaulting to grade 3; DeepSeek producing non-German output), establishing LLaMA 3.3 70b as the current SOTA for this specific task.
• Context Length Optimization: The study discovered that optimal context strategies differ by text length: shorter texts (Task 2) benefit from more examples (Range-of-Examples), while longer texts (Task 1) perform better with concise "Best-Average-Worst" contexts.

4. Results

The results indicate that while LLMs can utilize rubrics, they are not yet ready for fully autonomous deployment.

• Model Performance:

  • Mixtral and DeepSeek failed significantly (Mixtral assigned grade 3 to ~100% of texts; DeepSeek had high failure rates and output Chinese characters).
  • Qwen was too strict and lacked diversity.
  • LLaMA 3.3 70b was the only model capable of stable, diverse grading across the 1–5 scale.

• Accuracy Metrics:

  • Final Grade Agreement: The best result achieved was 32.8% exact match with human experts (using CoT with Best-Average-Worst prompting).
  • Sub-dimension Agreement: The maximum agreement for specific rubric dimensions (e.g., Content, Structure) was 40.6%.
  • QWK Scores: The highest QWK for the final grade was 0.46 (RAG-1-best), though few-shot methods showed higher QWK for specific dimensions (up to 0.55 for Task 1).

• Key Findings:

  • Few-shot prompting yielded the highest accuracy for final grades, particularly when using a mixed strategy for different text lengths.
  • Chain-of-Thought (CoT) improved exact accuracy (especially for short texts) but did not improve QWK, suggesting it might reduce the severity of errors without necessarily fixing the direction of errors.
  • RAG improved stability but failed to capture the full grading spectrum as effectively as few-shot learning.

5. Significance and Limitations

Significance:

• The study demonstrates that open-weight LLMs can effectively interpret complex, standardized rubrics for non-English languages, challenging the notion that only proprietary models (like GPT-4) can handle such tasks.
• It highlights that human oversight remains essential. The ~33% exact match rate indicates that LLMs are currently best suited as "co-graders" to reduce workload rather than replace human judgment.
• It provides a blueprint for context engineering in AES, showing that a "one-size-fits-all" context strategy is ineffective; strategies must be tailored to text length and type.

Limitations:

• Dataset Constraints: The dataset was limited to 101 texts (due to OCR artifacts from scanned/handwritten exams) and only covered 3 of the 7 possible A-level text types.
• Ground Truth: The dataset relied on a single human grader, introducing potential subjective bias into the "gold standard."
• Computational Cost: The best-performing model (LLaMA 3.3 70b) required significant resources (~240–750 seconds per task pair), making real-time application difficult without substantial infrastructure.
• Reproducibility: Due to computational limits, most results were based on single runs, lacking statistical variance analysis.

Conclusion:
The paper concludes that while fully automated, human-level grading is not yet achievable, LLMs combined with structured prompting and adaptive context strategies hold significant potential as supportive tools to enhance educator efficiency and fairness, provided that robust human oversight is maintained.