🤖 AI

SEFORA: Student Essays with Feedback Corpus and LLM Feedback Evaluation Framework

This paper introduces SEFORA, a large-scale corpus of student essays with real instructor feedback, and UniMatch, a novel evaluation framework that reveals current LLMs struggle to generate feedback aligning with human instructor priorities, achieving a maximum F1 score of only 0.4 across extensive experiments.

Original authors: Shayan Peyghambari Oskoui, Norah Almousa, Zhaoyi Joey Hou, Carolina Gustafson, Gayle Rogers, Raquel Coelho, Diane Litman, Xiang Lorraine Li

Published 2026-07-02

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shayan Peyghambari Oskoui, Norah Almousa, Zhaoyi Joey Hou, Carolina Gustafson, Gayle Rogers, Raquel Coelho, Diane Litman, Xiang Lorraine Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a teacher grading a stack of student essays. You have to read every sentence, find the good parts, point out the confusing parts, and write specific notes in the margins to help the student improve. It's hard work, and doing it for hundreds of students at once is nearly impossible for a human.

Recently, we've tried to use "AI teachers" (Large Language Models) to do this job. But there's a problem: we don't really know if the AI is giving good advice, and we don't have a good ruler to measure it.

This paper introduces two things to fix that: a massive library of real teacher notes called SEFORA, and a new measuring tool called UNIMATCH.

1. The Library: SEFORA (The "Gold Standard" Collection)

Think of SEFORA as a giant, organized scrapbook.

What's inside: It contains 564 drafts of student essays (some written once, some rewritten multiple times) from real college classes.
The special part: Next to every student essay, it has the actual notes written by real human professors. These aren't just red marks saying "wrong." They are sticky notes and highlights that say things like, "This character feels real," or "What does 'that' refer to here?"
Why it matters: Before this, most AI datasets only had scores (like "85/100") or simple error tags. SEFORA is special because it captures the nuance of how real teachers talk to students, including the specific parts of the text they are talking about.

2. The Ruler: UNIMATCH (The "Feedback Detective")

Now, imagine you ask an AI to write feedback on a student's essay. How do you know if it's any good?

The old way: You might compare the AI's words to the teacher's words to see if they look similar. But this is like judging a chef by whether they used the same words as the recipe, rather than if the food tastes right. If the teacher says "Make it shorter" and the AI says "Trim the fat," a simple word-count check might say they are different, even though they mean the same thing.
The new way (UNIMATCH): This tool acts like a detective. It breaks both the teacher's notes and the AI's notes down into tiny, individual "feedback units" (one specific thought per unit).
- Step 1: It splits the notes into pieces.
- Step 2: It asks, "Does this piece from the AI match the meaning of a piece from the teacher?" (e.g., Does "Trim the fat" match "Make it shorter"?).
- Step 3: It uses a smart matching system (like pairing socks) to see how many of the teacher's important points the AI managed to find and how many extra, useless points the AI invented.

3. The Big Discovery: The "Chatterbox" Problem

The researchers tested 74 different ways of asking AI models to write feedback. The results were surprising and a bit disappointing:

The Score: Even the smartest AI models only managed to get a score of about 0.4 out of 1.0. This means they are missing most of the specific, high-priority feedback a human teacher would give.
The Main Culprit: The biggest reason for the low scores was verbosity (talking too much).
- The Analogy: Imagine a student asks for help with one math problem. A good tutor gives one clear hint. The AI, however, acts like a nervous chatterbox. It gives the one hint, but then adds five more suggestions that the teacher never would have made, or repeats the same point three different ways.
- The Result: Because the AI generates so much "extra" noise, its precision drops. It's like a student who writes a 10-page essay to answer a one-sentence question; they might get the right answer, but they've buried it under so much fluff that it's not helpful.

Summary

The paper tells us that while AI can generate text, it currently struggles to act like a thoughtful human teacher. It tends to over-explain and miss the specific, high-value points that real instructors prioritize.

To fix this, we need:

Better Data: Real examples of how teachers actually talk (which SEFORA provides).
Better Measurement: A way to judge if the AI is hitting the right points, not just using the right words (which UNIMATCH provides).

Until AI learns to be concise and hit the right targets, it's not quite ready to replace the human touch in the classroom.

Technical Summary: SEFORA and UNIMATCH

Problem Statement

Effective writing feedback is a primary driver of student learning, yet producing it at scale is labor-intensive. While Large Language Models (LLMs) offer a pathway to automate this process, two critical bottlenecks hinder progress:

Data Scarcity: Few public corpora capture how instructors actually deliver feedback in real classrooms. Existing datasets often substitute structured labels (e.g., error tags, scores) for formative commentary, restrict coverage to single genres or prompts, or rely on crowd/AI-generated feedback rather than authentic instructor annotations.
Evaluation Deficiencies: Current methods for evaluating generated feedback are insufficient. Reference-free evaluation (e.g., LLM-as-a-judge) suffers from ambiguity and bias, as "good" feedback lacks a single operational definition. Reference-based metrics (e.g., BLEU, ROUGE, embedding similarity) fail to capture the semantic correspondence of open-ended feedback, particularly because feedback messages often bundle multiple distinct "feedback units" (e.g., a single sentence containing both praise and a critique). Holistic comparison cannot determine which specific observations are recovered or missed.

Methodology

The paper introduces two core components: the SEFORA corpus and the UNIMATCH evaluation framework.

1. SEFORA: Student Essays with Feedback Corpus

SEFORA is a public corpus designed to bridge the data gap. It pairs authentic, instructor-authored inline feedback with assignment prompts, rubrics, analytic scores, and multi-draft revisions across diverse college writing genres.

Scale and Scope: The corpus contains 564 drafts from 155 students across 34 assignments in 9 first-year undergraduate courses (covering composition, ESL, narrative, and creative writing). It includes 8,240 instructor annotations.
Annotation Granularity: Unlike error-tag datasets, SEFORA preserves the original paragraph structure and aligns annotations to specific text spans (phrases, sentences, or words). The feedback is predominantly "facilitative" (inviting reflection) rather than "directive" (prescribing changes), often posing open questions.
Privacy: Data collection was IRB-approved. Direct identifiers were removed, and names within essays were replaced with pseudonyms while preserving narrative coherence.

2. UNIMATCH: Reference-Based Evaluation Framework

UNIMATCH is a framework for evaluating LLM-generated feedback by comparing it against instructor feedback at the feedback unit level rather than holistically. The pipeline operates in three stages:

Feedback Segmentation:
- Feedback is segmented into "feedback units," defined as self-contained statements addressing one specific aspect of the writing.
- A segmentation guideline was developed, and an automatic segmenter (GPT-5-nano) was validated against human annotators, achieving high agreement (F1 = 0.97).
Feedback Unit Similarity Scoring:
- Pairs of instructor and model feedback units are scored on semantic correspondence using a 0–4 scale based on two dimensions: the target (aspect addressed) and the comment (evaluation/suggestion).
- A similarity scorer (gemini-3.1-flash-lite-preview) was trained/validated against human judgments, achieving a Pearson correlation of $r = 0.813$ .
- To reduce costs, a batching strategy was employed, which maintained high agreement with human scores.
Optimal Matching and Metric Calculation:
- UNIMATCH computes a maximum-weight bipartite matching (using the Hungarian algorithm) between the set of instructor units ( $G$ ) and model units ( $P$ ) based on their similarity scores.
- It calculates soft precision, soft recall, and F1 by normalizing the total matching weight against the self-similarity of each set. This approach captures both what the model omits (recall) and what it over-generates (precision).

Key Results

The authors evaluated 74 experimental configurations across multiple LLMs (including Llama-3.1-8B, Mistral-7B, DeepSeek-R1, Qwen2.5-72B, and GPT-5.1) using the UNIMATCH framework on the SEFORA corpus.

Performance Ceiling: No configuration exceeded an F1 score of 0.4. Even the strongest models (GPT-5.1 and Qwen2.5-72B) achieved only marginal improvements over smaller models, with GPT-5.1 reaching a precision of 0.38 and recall of 0.27.
The Verbosity Problem: The dominant factor driving poor alignment was verbosity. Models that generated more feedback units (over-generation) saw a sharp decline in precision without a compensatory increase in recall. The constrained prompt (limiting output to one unit) significantly outperformed unconstrained prompts.
Prompting Factors:
- Guidance: Providing explicit feedback categories (e.g., Task Constraints, Praise, Mistakes) improved F1 by approximately +0.059.
- Rubrics and Few-Shot: Including assignment rubrics had a negligible effect. Few-shot prompting provided a small gain (+0.017), primarily in precision, but was secondary to prompt structure and guidance.
End-to-End Validation: UNIMATCH scores correlated strongly ( $r \approx 0.84$ ) with expert human rankings of feedback quality, validating the framework's ability to recover instructor judgments.

Significance and Claims

The paper claims that the central challenge for LLMs in writing feedback is not the production of comments, but the prioritization of the specific comments an instructor would emphasize.

Pedagogical Relevance: The finding that over-generation harms performance is described as "pedagogically meaningful." Overwhelming students with excessive feedback is known to be counterproductive; thus, UNIMATCH's penalty for verbosity aligns with educational best practices.
Foundation for Future Work: By providing SEFORA and UNIMATCH, the authors lay a foundation for systematic study of LLM-generated feedback at the granularity in which instructors actually work. The framework offers a reliable, interpretable, and reference-based metric that moves beyond surface-level overlap to assess semantic alignment with instructor intent.
Limitations: The authors note that the evaluation operates on pre-segmented units (not span identification), relies on closed-source models for the evaluation pipeline (though open-source distillation is proposed as a next step), and treats instructor feedback as ground truth without a complementary human study of overall helpfulness or actionability.