💬 NLP

Exposía: Teaching and Assessment of Academic Writing Skills for Research Project Proposals and Peer Feedback

This paper introduces Expos'ia, the first public dataset linking student research proposals with multi-stage peer and instructor feedback, to benchmark large language models on automated scoring tasks and identify effective prompting strategies for educational assessment.

Original authors: Dennis Zyska, Alla Rozovskaya, Ilia Kuznetsov, Iryna Gurevych

Published 2026-04-16

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Dennis Zyska, Alla Rozovskaya, Ilia Kuznetsov, Iryna Gurevych

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are a student trying to write a proposal for a big research project. It's like trying to build a house without a blueprint: you have the bricks (ideas), but you don't know how to stack them so the roof doesn't fall in. Usually, you'd ask a teacher for help, but teachers are busy, and asking your classmates for help can be tricky because they might not know how to give good advice either.

This paper introduces Exposía, a new digital "playground" designed to teach computers how to help students with exactly this problem.

Here is the story of Exposía, broken down into simple parts:

1. The Problem: The "Feedback Loop" is Broken

In university, writing a research proposal (called an exposé) is the hardest part of learning to be a scientist.

The Challenge: Students struggle to write clear plans.
The Missing Piece: They also struggle to give good feedback to their peers.
The Result: Teachers are overwhelmed grading everything, and students don't get enough practice.

Think of it like a cooking class where everyone is trying to learn to make a soufflé. The teacher is too busy to taste every single dish, and the students are afraid to tell each other, "Hey, your oven is too hot," because they don't know how to say it nicely or correctly.

2. The Solution: A New Dataset (The "Recipe Book")

The researchers created Exposía, which is essentially a massive, organized library of student work. But it's not just a pile of essays. It's a time-traveling story for each student:

Draft 1: The student writes their first messy version of the proposal.
The Critique: They get feedback from two sources:
- Peers: Other students write comments and reviews.
- Teachers: Instructors write detailed comments and give scores.
The Revision: The student takes that feedback and writes a Final Version.

The Magic Ingredient:
Usually, datasets just have the final essay. Exposía has the whole journey. It links the messy draft, the specific comments (like "This sentence is confusing"), the final grade, and the improved final draft all together. It's like having a video recording of a chef fixing a bad dish, rather than just seeing the final plate.

3. The "Grading Rubric" (The Rulebook)

To make sure the data is useful, the researchers didn't just ask teachers to give a grade out of 10. They created a super-detailed Rulebook (called a rubric) with 36 specific rules for the proposal and 24 rules for the feedback.

For the Proposal: Did they state the problem clearly? Is the plan realistic? Is the bibliography correct?
For the Feedback: Did the student reviewer give actionable advice? Was their tone polite? Did they explain why something was wrong?

This is like a referee in a soccer game who doesn't just blow the whistle; they record exactly which rule was broken and how the player corrected it in the next play.

4. The Experiment: Can AI Be the Teacher?

The researchers asked: "Can modern AI (Large Language Models) learn to grade these proposals and feedback as well as human teachers?"

They tested the AI on two jobs:

Grading the Proposal: "Is this research plan good?"
Grading the Feedback: "Is this student's review helpful?"

The Results:

The "Super-Teachers" (Closed-Source AI): The most powerful, expensive AI models (like the ones from big tech companies) did a great job. They were very close to human teachers.
The "Local Teachers" (Open-Source AI): The free, downloadable AI models were okay, but they made more mistakes. This is a problem because schools often can't afford the expensive ones or can't send student data to the cloud for privacy reasons.
The "One-Size-Fits-All" Mistake: The researchers found that the AI needed different "personalities" for the two jobs. An AI that was great at grading the proposal was sometimes terrible at grading the feedback, and vice versa.

5. The Big Takeaway: How to Use AI in Class

The paper concludes with a few important lessons for the future of education:

Don't just ask the AI for one score: It works best if you ask it to look at all the rules at once (like a holistic judge) rather than checking one rule, then another, then another.
AI is a "Co-Pilot," not the Captain: The AI is great at spotting patterns and suggesting grades, but it shouldn't replace the human teacher. Sometimes the AI misses the "soul" of the research or gets confused by tricky wording. The best approach is Human-in-the-Loop: The AI does the heavy lifting, and the human teacher makes the final call.
Privacy Matters: Since schools can't always send student work to the cloud, we need to build better, free AI models that can run on school computers to protect student privacy.

In a Nutshell

Exposía is a new tool that helps computers learn how to teach writing and feedback. It proves that while AI is getting very good at grading, it still needs human guidance, and we need to build better, private, and free AI tools so every student can get the help they need to write their next big idea.

1. Problem Statement

The paper addresses two critical challenges in higher education:

Research Proposal Writing: Writing a research project proposal (exposé) is a primary hurdle for students, requiring skills in formulating research questions, defining methodologies, and structuring arguments. Many students lack these competencies.
The Peer Review Crisis: While peer feedback is a powerful pedagogical tool, students are rarely explicitly taught how to provide high-quality feedback. Furthermore, the manual evaluation of both proposals and peer feedback is labor-intensive for instructors, creating a bottleneck in scalable education.

Existing Natural Language Processing (NLP) resources are fragmented. Automated Essay Scoring (AES) datasets typically focus on general persuasive essays with holistic scores, lacking the granular feedback, revision history, and specific domain constraints of research proposals. Conversely, peer review datasets often focus on professional scientific communication rather than educational learning outcomes. There is a lack of a unified dataset connecting drafting, feedback (peer/instructor), revision, and fine-grained assessment scores.

2. Methodology and Dataset Construction (Exposía)

The authors introduce Exposía, the first public dataset connecting writing and feedback in a higher education setting.

Data Collection Context: The data was collected from the "Introduction to Scientific Work" course (Bachelor's level, Computer Science) at TU Darmstadt in Winter 2024.
Workflow:
1. Drafting: Students write a draft exposé on one of three AI-related topics.
2. Feedback: Each draft receives double-blind peer reviews from two students and instructor reviews. Feedback consists of:
  - Inline Comments: Span-anchored comments with tags (Strength, Weakness, Highlight, Other).
  - Free-text Reviews: Document-level summaries.
3. Revision: Students revise their drafts based on feedback to produce a final exposé.
Annotation Schema:
- Exposé Assessment: 36 fine-grained criteria organized into 9 rubrics (e.g., Motivation, Approach, Methodology, Schedule). Criteria are rated on ordinal scales (e.g., 0–2).
- Review Assessment: 24 fine-grained criteria organized into 8 rubrics (e.g., Feed Up/Back/Forward, Content Quality, Actionability).
- Inter-Annotator Agreement (IAA): A subset of data was graded by two independent instructors to calculate IAA (using Krippendorff's $\alpha$ , Pearson $r$ , and Quadratic Weighted Agreement).
Dataset Statistics: The initial release contains 55 exposés, 306 reviews, and 2,253 inline comments.

3. Key Contributions

Novel Dataset: Exposía is the first dataset to integrate drafting, multi-source feedback (peer/instructor), revision, and fine-grained criterion-based scoring for research proposals.
Pedagogically Grounded Rubrics: The authors developed a comprehensive, topic-independent assessment schema (36 criteria for proposals, 24 for reviews) derived from cross-disciplinary university guidelines and educational theory (e.g., Hattie & Timperley's feedback model).
Benchmarking LLMs: The paper establishes baselines for Large Language Models (LLMs) on two distinct tasks:
- Task 1: Automated scoring of research proposals.
- Task 2: Automated scoring of student peer reviews.
Prompting Strategy Analysis: The study compares single-criterion prompting (scoring one criterion at a time) vs. combined prompting (scoring all criteria in one call).

4. Experimental Results

The authors evaluated state-of-the-art LLMs (both closed-source like GPT-5.2, Claude Opus, Gemini, and open-weight models like Llama 3.3, Qwen, GPT-OSS) using binned Jensen-Shannon Divergence (JSD) to measure alignment with human score distributions.

Prompting Efficiency: Combined prompting significantly outperformed single-criterion prompting in both alignment with human raters and computational efficiency (reducing runtime by 1.3x–5.3x). This is crucial for classroom deployment where deadlines create peak loads.
Model Performance:
- Closed-source models (e.g., GPT-5.2, Claude Opus) consistently outperformed open-weight models.
- Task Specificity: Different models excelled at different tasks. For instance, GPT-5.2 was best for proposal scoring but poor for review scoring, while Llama 3.3 showed the opposite trend.
- Open-Weight Gap: Open-weight models struggled more with the nuance of feedback scoring, highlighting a need for better privacy-preserving, locally deployable models for education.
Human-LLM Alignment: While models showed reasonable alignment, they exhibited systematic biases. For example, some models failed to detect the absence of a clear research question (a critical flaw) that human instructors caught.
Revision Impact: Analysis showed that students improved significantly in "Approach," "Motivation," and "Methodology" after receiving feedback, validating the pedagogical value of the feedback loop.

5. Significance and Future Directions

Educational Impact: Exposía enables research into formative assessment and feedback-mediated learning. It allows for the study of how specific types of feedback (e.g., actionable vs. vague) lead to specific improvements in student writing.
Scalability: The findings suggest that while LLMs can assist in grading, they should function as decision-support tools (human-in-the-loop) rather than fully automated graders, especially given the variability in human scoring and the risk of "gaming" the system.
Privacy and Deployment: The study underscores the necessity of open-weight models for educational settings due to data privacy constraints, motivating future work to improve the performance of locally deployable models.
Broader Applicability: Although collected in Computer Science, the assessment framework is designed to be generalizable to other disciplines, providing a template for evaluating research proposal writing and peer feedback skills across academia.

In conclusion, Exposía bridges the gap between computational linguistics and educational science, providing a rich resource to develop AI tools that not only grade but actively support the complex process of learning to write and critique scientific research.