💬 NLP

Mitigating Bias in Automated Grading Systems for ESL Learners: A Contrastive Learning Approach

This study addresses algorithmic bias in automated essay scoring against ESL learners by employing a contrastive learning approach with matched essay pairs, which successfully reduced high-proficiency scoring disparities by 39.9% while maintaining overall grading accuracy.

Original authors: Kevin Fan, Eric Yun

Published 2026-01-26

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Kevin Fan, Eric Yun

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Strict Teacher" with a Bias

Imagine you have a very smart robot teacher designed to grade essays automatically. This robot has read thousands of essays written by native English speakers and learned what a "good" essay looks like based on that experience.

The researchers found a problem: When this robot grades essays written by students who speak English as a second language (ESL), it gets confused. Even if an ESL student writes a brilliant, high-quality essay, the robot often gives them a lower score than a native speaker who wrote an essay of the exact same quality.

Why does this happen?
The robot is taking a "shortcut." Instead of reading the deep meaning of the story (the semantics), it is looking at surface-level clues, like sentence length or specific grammar patterns.

The Analogy: Imagine a judge who thinks, "If a sentence is long and complex, it must be a mistake." Native speakers often write long, complex sentences naturally. But ESL students might write long, complex sentences because they are trying hard to express a complex idea in a second language. The robot sees the length and thinks, "This looks like an error," and lowers the score. It's like a music critic who hates jazz because the notes don't follow the rules of classical music, even though the jazz is beautiful.

The Discovery: The "Score Ceiling"

The researchers tested a top-tier AI model (DeBERTa) and found a specific "ceiling" for ESL students.

If a native speaker wrote a perfect essay, the robot gave them a high score (e.g., 9/10).
If an ESL student wrote an essay that humans rated as equally perfect, the robot capped their score lower (e.g., 8/10).
The Result: The robot was systematically underestimating high-proficiency ESL writers by about 10%, even though the essays were objectively just as good.

The Solution: The "Twin Training" Method

To fix this, the researchers didn't just tell the robot to "be fair." Instead, they changed how they trained it using a technique called Contrastive Learning.

The Analogy: The "Twin" Exercise
Imagine you are training a new coach to judge athletes.

The Old Way: You show the coach a native athlete and say, "This is a gold medal." Then you show an ESL athlete and say, "This is a silver medal." The coach learns that "Native = Gold" and "ESL = Silver," regardless of actual performance.
The New Way (Triplet Strategy): The researchers created a special training set with groups of three essays (Triplets):
- Anchor: A native speaker's essay with a score of 9.
- Positive (The Twin): An ESL essay that humans also rated as a 9.
- Negative (The Mismatch): An essay (native or ESL) that was rated as a 5.

The robot was forced to learn a specific rule: "The Native essay and the ESL essay (the Twins) must look exactly the same to you because they have the same quality. The Mismatched essay must look very different."

By forcing the robot to see the ESL essay and the Native essay as "twins" in terms of quality, the robot learned to ignore the "accent" (surface grammar quirks) and focus on the "soul" (the actual quality of the writing).

The Results: Fairer Without Losing Smarts

After this new training, the researchers tested the robot again:

Bias Reduced: The gap between how the robot scored native vs. ESL students dropped from 10.3% down to 6.2%. That's a 40% reduction in unfairness.
Accuracy Kept: The robot didn't get "dumber." It still agreed with human graders about 76% of the time (a score known as QWK), which is considered very good for this type of work.
What Changed: The robot stopped penalizing ESL students for using complex sentence structures. It learned that a long, complex sentence in an ESL essay is a sign of effort and skill, not a mistake.

The Catch (Limitations)

The paper notes a few things to keep in mind:

Conservative Grading: The new robot became slightly more "cautious." It gave slightly lower scores to everyone (both native and ESL) compared to before. It fixed the gap between the groups, but the absolute scores might need a little more tuning to be perfect.
Specific to This Model: This fix was tested on one specific type of AI (DeBERTa) and for English learners. It might need to be retrained to work on other languages or different AI models.

Summary

The researchers built a "fairness training camp" for an AI grader. By showing the AI that a native essay and an ESL essay can be "twins" in quality, they taught the AI to stop judging students based on their "accent" (surface grammar) and start judging them based on their actual ideas. The result is a grading system that is much fairer to ESL students without losing its ability to grade accurately.

Technical Summary: Mitigating Bias in Automated Grading Systems for ESL Learners

Problem Statement
As Automated Essay Scoring (AES) systems, particularly those based on Transformer architectures like BERT and DeBERTa, are deployed in high-stakes educational settings, they exhibit significant algorithmic bias against English as a Second Language (ESL) learners. The paper identifies that these models often engage in "shortcut learning," relying on surface-level linguistic heuristics rather than deep semantic reasoning. Specifically, Transformer attention heads disproportionately attend to distinct L2 markers (e.g., prepositional misuse, tense inconsistencies) as proxies for low quality. This results in a "constrained scoring effect" where high-proficiency ESL essays are systematically under-scored compared to Native speaker essays of identical human-rated quality. The authors' initial analysis of a fine-tuned DeBERTa-v3 model revealed that high-proficiency ESL essays received scores approximately 10.3% lower than their Native counterparts, despite having equivalent human ratings.

Methodology
To address this representation gap without sacrificing overall grading accuracy, the authors propose a domain-specific intervention: Contrastive Learning with Matched Essay Pairs.

Data Construction: The study utilized a unified benchmark merging the ASAP 2.0 dataset (containing both Native and ESL essays) and the ELLIPSE dataset (ESL-specific). Scores from both datasets were normalized to a [0, 1] range.
Triplet Formation: The core of the methodology involves algorithmically generating "Fairness Triplets" $(A, P, N)$ $(A, P, N)$ :
- Anchor (A): A Native speaker essay with a normalized score $S$ .
- Positive (P): An ESL essay with a human score within a tight range of the anchor ( $[S - 0.02, S + 0.02]$ ).
- Negative (N): An essay (Native or ESL) with a score significantly different from the anchor ( $[S \pm 0.20]$ ).
  This construction forces the model to treat ESL and Native essays of equivalent quality as "quality twins," mathematically requiring their latent embeddings to be closer to each other than to essays of different quality.
Model Architecture & Training:
- Base Model: Microsoft's DeBERTa-v3-base, fine-tuned using Low-Rank Adaptation (LoRA) with rank $r=16$ .
- Loss Function: The regression head was removed, and the backbone was fine-tuned using Triplet Margin Loss: $L = \max(0, d(A, P) - d(A, N) + \alpha)$ , where $d$ is Euclidean distance and $\alpha$ is the margin.
- Hyperparameters: Preliminary ablation studies determined an optimal margin $\alpha = 1.0$ . The model was trained for 2 epochs to align the latent space while preserving pre-trained knowledge.
- Scoring: After alignment, the backbone was frozen, and a lightweight linear regression head was trained for 5 epochs to map the fair embeddings back to scalar scores.

Key Results
The proposed approach demonstrated a significant reduction in bias while maintaining acceptable grading accuracy:

Bias Reduction: The scoring disparity (gap) for high-proficiency essays was reduced from 10.3% in the baseline model to 6.2% in the contrastive model, representing a 39.9% reduction in the bias gap.
Accuracy Maintenance: The model achieved a Quadratic Weighted Kappa (QWK) of 0.756, which remains above the 0.70 threshold typically required for "substantial agreement" in educational assessment. This indicates the model learned a fairer representation without collapsing the embedding space or severely degrading performance.
Ablation Findings: An "aggressive" setting with $\alpha = 2.0$ reduced the bias gap slightly further but caused a more significant drop in QWK (0.718), suggesting that $\alpha = 1.0$ represents the optimal Pareto frontier between fairness and accuracy.
Linguistic Analysis: Post-hoc analysis using spaCy and textstat revealed that the baseline model incorrectly penalized sentence complexity in ESL writing, mistaking valid L2 syntactic structures for errors. The contrastive model successfully disentangled sentence complexity from grammatical errors, ceasing to penalize stylistic features common in high-proficiency L2 writing.

Significance and Claims
The paper claims that the traditional trade-off between fairness and accuracy in AES may be less severe than previously assumed. By utilizing matched triplet pairs, the authors demonstrate that contrastive learning can effectively target specific latent representations of quality rather than relying on broad adversarial techniques or data augmentation, which often degrade model performance.

The study concludes that this approach offers a viable framework for mitigating demographic bias in encoder-based models. However, the authors remain modest regarding immediate deployment, noting that the model currently exhibits a systematic under-scoring trend for both groups (suggesting a more conservative scoring policy) and that the model requires retraining to generalize to other demographics or architectures (e.g., RoBERTa, ELECTRA). Future work is proposed to explore score recalibration techniques and multi-task learning to further refine the balance between fairness and absolute accuracy.