Technical Summary: Cross-Rubric Generalization for Critical Thinking Essay Scoring

Problem Definition

Automated Essay Scoring (AES) research has traditionally focused on cross-prompt generalization, where models score essays from unseen prompts while adhering to a constant set of scoring criteria (rubrics). However, real-world educational assessment often requires cross-rubric generalization: training a model on essays labeled under one set of rubrics and evaluating its ability to score essays under previously unseen rubrics that target different aspects of writing. This is particularly challenging in critical thinking assessment, where new rubric criteria may be introduced over time, and obtaining expert annotations for these new rubrics is expensive.

The core challenge is that the model must generalize to new scoring criteria where the relevant textual evidence, its interpretation, and the definition of proficiency differ from the training distribution. This paper defines the task as mapping essays to discrete proficiency levels defined by natural-language rubrics, where the model is trained on a disjoint set of training rubrics ($\mathcal{R}_{train}$) and essays ($\mathcal{E}_{train}$) but must evaluate essays ($\mathcal{E}_{test}$) under unseen test rubrics ($\mathcal{R}_{test}$).

Methodology

The authors propose a Large Language Model (LLM) fine-tuning framework that addresses cross-rubric generalization through two primary components: rubric-agnostic intermediate representations (traits) and target-essay supervision strategies.

1. Rubric-Agnostic Trait Representations

To bridge the gap between specific rubrics and general essay quality, the authors introduce a set of six analytic traits grounded in argumentation and critical thinking theory. These traits serve as a shared intermediate representation ($h(e)$) that captures lower-level properties recurring across different rubrics:

• Claim Explicitness: Clarity of the main point.
• Structural Coherence: Organization and connection of ideas.
• Supporting Evidence: Use of examples and facts.
• Evidence–Claim Linkage: Explanation of how evidence supports the claim.
• Alternative Perspectives: Consideration of other viewpoints.
• Analytical Depth: Reasoning beyond description.

These traits are defined as fixed, rubric-agnostic categories (e.g., absent/partial/strong). The framework explores four configurations for integrating these traits:

• Input-only: Appending trait values (and optionally natural language observations) to the model input.
• Output-only: Training the model to jointly predict trait values (and observations) alongside the final rubric label.

2. Supervision Settings

The study investigates how the availability of essay labels during training affects generalization to unseen rubrics, considering three distinct settings:

• No Labels (Hardest): The model is trained only on $\mathcal{E}_{train}$ and $\mathcal{R}_{train}$. Target essays ($\mathcal{E}_{test}$) are unseen during training.
• Pseudo Labels: The model is first trained in the "No Labels" setting, then used to generate pseudo-labels for $\mathcal{E}_{test}$ under seen rubrics ($\mathcal{R}_{train}$). A new model is trained on this augmented dataset.
• Gold Labels: Target essays ($\mathcal{E}_{test}$) are available during training but only under seen rubrics ($\mathcal{R}_{train}$) with human-annotated labels. The model is evaluated on $\mathcal{E}_{test}$ under unseen rubrics ($\mathcal{R}_{test}$).

3. Model Architecture

The authors fine-tune Llama 3.1 8B Instruct using a conditional text generation approach. For each essay-rubric pair, the model generates a natural-language justification followed by the rubric label. The training objective minimizes cross-entropy loss.

Key Results

Experiments were conducted on a dataset of 500 student critical thinking essays annotated under six rubrics. The results highlight the following trends:

Impact of Traits and Supervision

• Traits in the Hardest Setting: In the No Labels setting (no target-essay supervision), incorporating traits significantly improves performance. Specifically, the "Output: Trait Values" configuration improved Macro F1 by 5.0% (from 0.353 to 0.403) and Accuracy by 3.7% compared to the baseline without traits. Predicting traits as outputs proved more effective than providing them as inputs, suggesting that generating trait-level abstractions encourages structured mapping.
• Diminishing Returns with Supervision: The benefit of traits diminishes as supervision increases. In the Gold Labels setting, trait-based configurations did not consistently outperform the baseline without traits, suggesting that direct supervision on target essays allows the model to learn rubric-agnostic representations more effectively without intermediate trait scaffolding.
• Supervision Impact: Increasing supervision from No Labels to Gold Labels consistently improved performance. The Gold Labels setting yielded the strongest gains, improving Macro F1 by 7.8% over the No Labels baseline. Pseudo Labels offered a competitive, practical alternative when gold labels are unavailable.

Comparison with Baselines

• Fine-tuning vs. Zero-Shot: Fine-tuning the open-source Llama model substantially outperformed zero-shot prompting of the same model. The F1 score increased from 0.222 (zero-shot) to 0.353 (fine-tuned baseline), demonstrating that exposure to the label distribution during training is critical for calibration and ordinal alignment.
• Open-Source vs. Proprietary: The best fine-tuned model (Gold Labels, No Traits) achieved a Macro F1 of 0.431. This outperformed the proprietary GPT-5-mini (0.410) by 2.1% and trailed the larger GPT-5 (0.450) by only 1.9%.

Significance and Contributions

The paper claims three primary contributions:

1. Problem Formulation: It establishes cross-rubric generalization as a distinct and critical challenge in AES, moving beyond the standard cross-prompt setting to address scenarios where scoring criteria themselves change.
2. Systematic Analysis: It provides a systematic study of two key mechanisms for improving generalization: rubric-agnostic trait representations and target-essay supervision.
3. Practical Efficacy: The results demonstrate that these components improve generalization in complementary ways. Traits are particularly vital in low-supervision settings (No Labels), while strong supervision enables fine-tuned open-source models to rival proprietary prompting-based systems.

The authors conclude that trait-based intermediate structures and controlled supervision offer a scalable, cost-effective, and privacy-preserving alternative for rubric-based assessment, particularly as scoring criteria evolve in educational settings. They note that future work should examine the generalizability of these findings to other datasets and writing domains, and explore stronger low-supervision learning techniques.