Technical Summary: Pair2Score

Problem Statement

Many evaluation applications, particularly Automated Essay Scoring (AES), require absolute predictions (e.g., assigning a specific score to an essay based on a rubric). However, collecting absolute labels is often resource-intensive, whereas pairwise comparisons (determining which of two essays is better) are easier to obtain and may carry stronger local information.

The central research question addressed by this paper is: Can pairwise learning serve as an effective intermediate objective for rubric-aligned absolute essay scoring? Specifically, the authors investigate how to transfer knowledge from a comparative model trained on document pairs into a downstream absolute scorer, using Large Language Models (LLMs).

Methodology: Pair2Score Framework

The authors propose Pair2Score, a two-stage learning framework designed to transfer pairwise comparisons into absolute scoring using parameter-efficient LLM adaptation (LoRA). The framework utilizes a Meta LLaMA-3.2-1B backbone and operates as follows:

Stage 1: Directional Siamese Ranking (Relative Objective)

• Architecture: A Siamese setup with a shared LLaMA+LoRA backbone processes two documents ($a$ and $b$).
• Objective: The model learns a directional preference. It uses a bias-free linear utility head to compute scalar utilities $s(h_a)$ and $s(h_b)$ from pooled hidden representations. The comparison logit is defined as $\Delta(a, b) = s(h_a) - s(h_b)$.
• Constraint: The design enforces antisymmetry ($\Delta(b, a) = -\Delta(a, b)$), ensuring consistent preferences regardless of input order.
• Loss: The model is trained using pairwise logistic loss to maximize $\Delta(a, b) > 0$ for pairs where document $a$ has a higher absolute trait label than $b$.
• Output: The stage produces a trained LoRA adapter and, optionally, pooled embeddings.

Stage 2: Absolute Scoring with Transfer (Absolute Objective)

• Goal: Train an absolute regressor to predict trait scores (e.g., grammar, vocabulary, syntax) directly, optimizing L1 loss and evaluating with Quadratic Weighted Kappa (QWK).
• Transfer Mechanisms: The authors compare three configurations:
  1. Baseline (Absolute-only): A fresh LoRA adapter is trained from scratch on absolute labels.
  2. Warm-start: The absolute scorer is initialized with the LoRA adapter weights from Stage 1 (the pairwise head is discarded).
  3. Embedding Fusion: In addition to warm-start initialization, precomputed Stage 1 embeddings are concatenated with the current pooled representation before the regression head.
• Variables: The study varies the pair-set size (small vs. large) and the pairwise-stage duration (standard multi-epoch vs. a single epoch).

Experimental Protocol

• Dataset: The Feedback Prize English Language Learning dataset (ELLIPSE corpus), focusing on three traits: grammar, vocabulary, and syntax.
• Evaluation: A five-fold cross-validation protocol where the held-out fold and the random seed are co-rotated (e.g., Fold A uses seed 42, Fold B uses seed 48). This ensures stochasticity is part of the measured evidence rather than averaged out.
• Metrics: Primary metric is test QWK.

Key Contributions

1. Design Space Formulation: The paper frames relative-to-absolute scoring transfer as a design space defined by three axes: intermediate objective, transfer mechanism, and downstream target. It instantiates one specific path (pairwise preference $\to$ warm-start/fusion $\to$ absolute regression).
2. Empirical Findings on Transfer: Across 15 fold-level evaluations (3 traits $\times$ 5 folds), the authors find that pairwise ranking quality alone does not predict whether transfer will improve absolute scoring. Instead, the choice of transfer mechanism and the duration of the pairwise stage are critical.
3. Duration Sensitivity: A clear pattern emerges regarding pairwise-stage duration: a single epoch of pairwise training transfers more reliably than extended training. This suggests the pairwise stage functions primarily as an effective early initialization rather than a prolonged ranking optimization phase.
4. Lack of Correlation: The study demonstrates a weak correlation between Stage 1 pairwise accuracy and Stage 2 absolute scoring performance, indicating that downstream success depends more on the transfer configuration and fold/seed variations than on marginal gains in Stage 1 ranking metrics.

Results

• Performance: The best-performing transfer variant improved QWK over the absolute-only baseline for all three traits (Grammar, Vocabulary, Syntax).
• Consistency: No single configuration dominated across all runs. However, when the best transfer variant was selected for each trait-fold evaluation, pairwise transfer outperformed the baseline in 11 of 15 cases, with a mean $\Delta$QWK of +0.012.
• Configuration Impact:
  • Duration: The "one-epoch" pairwise setting consistently outperformed or tied with the "standard" (multi-epoch) setting across most configurations.
  • Pair-set Size: Increasing the number of pairs (from ~3k to ~6k) did not yield a consistent advantage.
  • Transfer Mechanism: Both warm-start and embedding fusion showed benefits, but the specific "winner" varied by trait and fold.

Significance and Claims

The paper modestly claims that Pair2Score demonstrates that comparative supervision can be a useful intermediate stage for rubric-aligned absolute scoring, but only if the transfer pathway is carefully configured.

The authors emphasize that the benefit is not automatic; simply adding a pairwise stage does not guarantee improvement. The key insight is that the pairwise stage contributes primarily through early initialization (providing a better starting point for the absolute scorer) rather than through the accumulation of ranking knowledge over many epochs.

The work positions relative-to-absolute transfer as a distinct modeling question where the intermediate objective, transfer pathway, and downstream target must be considered together. The authors acknowledge limitations, noting that their evaluation covers only one path through the broader design space (specifically, pairwise preference with warm-start/fusion in an AES setting) and that further validation is needed for other intermediate objectives, backbones, and assessment workflows.