Training data generation for context-dependent rubric-based short answer grading

1. Problem Statement

The paper addresses the critical bottleneck in Natural Language Processing (NLP) research: the lack of access to high-quality, domain-specific data due to privacy regulations and non-disclosure agreements (NDAs).

• Specific Context: The authors aim to train models for Short Answer Grading (SAG), specifically for the Programme for International Student Assessment (PISA) test.
• The Challenge: The PISA dataset contains valuable data (context, questions, grading rubrics, student answers, and ratings) but is confidential. Researchers cannot publish or share it directly.
• The Dilemma:
  • Using the raw confidential data for training violates NDAs.
  • Generating entirely synthetic data often results in a distribution that is too dissimilar to the real data to be useful for training robust evaluators.
• Goal: Create an open, "surrogate" training dataset that mimics the statistical and semantic properties of the confidential PISA dataset without revealing any individual datapoints or violating the NDA.

2. Methodology

The authors propose a pipeline to generate a large-scale proxy dataset and then select the most representative subset using Derived Text Formats (DTFs) and feature matching.

A. Reference Dataset

The target distribution is based on a confidential PISA subset containing:

• Context: Background text.
• Question: Open-ended query based on context.
• Grading Rubric: Criteria for Full, Partial, and No credit.
• Answer: Student response.
• Rating: Ternary label (Full/Partial/No credit).

B. Synthetic Candidate Generation

A multi-stage pipeline using Large Language Models (LLMs) and open-source web data was used to create a massive pool of synthetic candidates:

1. Context Extraction: Plain text was scraped from open websites, tokenized, and concatenated into chunks (150–800 words). LLMs cleaned these texts of impurities.
2. Question Generation: LLMs generated 1–5 open-ended questions per context, filtered by a "judge" LLM to ensure they were answerable, logical, and objective.
3. Rubric Generation: A 2-stage process generated rubrics for Full, Partial, and No credit. A judge LLM iteratively refined the rubrics until they met strict criteria (no ambiguity, distinct sections) and matched the style of OECD examples.
4. Answer Generation: LLMs generated 5 candidate answers per rubric level (Full/Partial/No). A judge LLM verified that answers adhered to the specific instructions for their correctness level (e.g., partial credit answers must show plausible misinterpretation).

C. Feature Extraction

To align the synthetic data with the confidential data without exposing the raw text, the authors extracted 14 linguistic and semantic features for every datapoint, including:

• Embedding Similarities: Cosine similarity between Context/Question, Context/Answer, and Rubric/Answer using the BAAI/bge-m3 encoder.
• N-gram Metrics: Recall and Jaccard similarity for unigrams and bigrams between questions, contexts, answers, and rubrics.
• Statistical Features: Answer length, lexical density, and TF-IDF cosine similarities.

D. Data Selection Strategies

Three methods were tested to select the best subset from the synthetic pool to match the distribution of the confidential data:

1. Method 1 (Global Mean): Selected the top 5% of samples closest to the global mean of the confidential dataset's features (L2 metric).
2. Method 2 (Clustered Mean): Addressed domain coverage issues by clustering the confidential data (k-means) and selecting the top 5% closest to the nearest of 8 cluster means.
3. Method 3 (Rank-Based): Relaxed the "no information leakage" constraint slightly by using unreduced features. It ranked synthetic samples based on their L2 distance to the closest 5 reference samples, averaging the rank indices to determine selection.

3. Key Contributions

• Privacy-Preserving Data Synthesis: Demonstrated a workflow to generate a large-scale training dataset for a restricted domain (PISA) using only a small confidential reference set and open-source data.
• Derived Text Formats (DTFs): Proposed a novel approach to data selection using high-level linguistic features rather than raw text, ensuring that individual confidential datapoints cannot be reconstructed from the generated set.
• Comparative Evaluation: Systematically compared three selection strategies to determine which best preserves the distributional properties of the reference data for model training.

4. Results

The authors evaluated the selected datasets using three experimental setups (E1: Few-shot accuracy, E2: Model selection proxy, E3: Full model training):

• Method 1 & 2 (Reduced Features):
  • Feature Matching: Improved similarity in some features (e.g., answer length) but failed in others (e.g., bigram recall for no-credit answers).
  • Performance: Both methods failed to improve model performance. In E1 and E2, models trained on these subsets performed worse or similarly to random baselines. The authors attribute this to the reduced features being too reductive to capture the necessary complexity for model selection.
• Method 3 (Rank-Based/Unreduced Features):
  • Performance: This was the only successful method.
  • E3 Results: A model trained on the Method 3 subset achieved a statistically significant improvement over a model trained on a random subset.
  • Statistical Significance:
    • Initial run: $p = 0.00091$ (Average gain: +0.0206 accuracy).
    • Repeated runs: $p = 0.00102$ (Average gain: +0.0116 accuracy).
  • Conclusion: The rank-based selection using unreduced features successfully identified a subset that improved training stability and final accuracy.

5. Significance and Future Work

• Implication: The study proves that it is possible to create effective training data for restricted domains without violating NDAs, provided the selection method is sophisticated enough to capture the underlying data distribution.
• Limitations: The current evaluation relies on a single training run and a specific model architecture. The authors note that simple, averaged features (Methods 1 & 2) are often insufficient; more expressive feature sets or complex weighting are likely needed for broader applicability.
• Future Directions: The authors plan to iterate on subset selection methods using more expressive features and test the approach across a wider variety of models and datasets to confirm generalizability.

In summary, the paper successfully navigates the tension between data privacy and model training needs. While simple feature-matching strategies failed, a more nuanced, rank-based selection approach using unreduced features yielded statistically significant improvements in model training, offering a viable path forward for research in restricted domains like educational assessment.