Implicit Grading Bias in Large Language Models: How Writing Style Affects Automated Assessment Across Math, Programming, and Essay Tasks

This study demonstrates that despite explicit instructions to evaluate only content correctness, large language models exhibit significant, subject-dependent grading bias against informal and non-native writing styles in essay tasks, while showing minimal bias in mathematics and programming assessments.

The Gist

Imagine you have a super-smart robot teacher named "GraderBot." This robot can read thousands of essays, math problems, and coding assignments in the blink of an eye. Schools are excited because this robot can save teachers hours of work.

But here's the catch: Is GraderBot fair?

This paper asks a simple but scary question: If two students know the exact same answer, but one writes it like a professor and the other writes it like a casual friend, will the robot give them the same grade?

The researchers set up a "trap" to find out. They took 180 correct answers and secretly changed how they sounded without changing what they meant. They created three types of "style traps":

1. The "Sloppy" Trap: Adding typos and bad grammar.
2. The "Chill" Trap: Using slang and casual words (like saying "u gotta" instead of "you must").
3. The "Foreign" Trap: Writing in a way that sounds like someone whose first language isn't English (even if the grammar is technically okay).

Then, they asked two of the world's smartest AI robots (LLaMA and Qwen) to grade these answers. They even told the robots: "Hey, ignore the style! Only grade the correct answer!"

Here is what happened, explained simply:

1. The "Math & Code" Zone: The Robot is Fair

When the students answered Math or Programming questions, the robot was a perfect judge.

• The Analogy: Imagine a robot checking if a lock is open. If the door is open, it's open. It doesn't matter if the person who opened the door was wearing a tuxedo or a dirty jumpsuit. The result is the same.
• The Result: Whether the student wrote "2x = 8" formally or "u gotta subtract 5 to get x=4" casually, the robot gave them full marks. The code either worked, or it didn't. The math was right, or it wasn't.

2. The "Essay" Zone: The Robot is Biased

When the students wrote Essays, the robot went crazy. It started punishing students for how they sounded, even though they were told to ignore it.

• The Analogy: Imagine a food critic who is supposed to judge a burger only by how it tastes. But, if the burger is served on a fancy plate, they give it 10/10. If the same burger is served on a napkin, they give it a 6/10, claiming it "lacks quality." The taste (the content) is identical, but the presentation (the style) ruined the score.
• The Result:
  • Students who used slang/informal language got hit the hardest. The robot deducted nearly 2 points out of 10. That's the difference between a B+ and a C+.
  • Students who sounded non-native also got penalized, losing about 1 point.
  • Even students with grammar mistakes got a small penalty.

3. The "Magic Spell" Didn't Work

The researchers tried to "fix" the robot by giving it a magic spell (a prompt instruction): "Do NOT penalize for style!"

• The Analogy: It's like telling a dog, "Don't chase that squirrel!" while pointing at the squirrel. The dog knows the rule, but its brain is wired to chase squirrels. The robot's brain was trained on millions of formal books and articles. It learned that "formal writing = smart" and "casual writing = sloppy." Even when told to stop, its brain couldn't unlearn that connection.

Why Does This Matter?

This isn't just about a few points on a test. It's about fairness.

• The Real World: Many students are brilliant but don't write like professors. Maybe they grew up speaking a different language, maybe they are from a culture where talking casually is normal, or maybe they just think differently.
• The Danger: If schools start using these robots to grade essays, these smart students will get lower grades not because they are dumb, but because their "voice" doesn't match the robot's training. It's like a race where everyone has to run the same distance, but some runners are forced to wear heavy boots while others wear sneakers.

The Bottom Line

The paper concludes that AI grading is great for Math and Coding, where the answer is black and white. But for Essays and Writing, the AI is currently too biased to be trusted alone.

The Recommendation:
Before schools let robots grade essays, they need to:

1. Test the robot with different writing styles to see if it's biased.
2. Keep humans in the loop for anything that requires a "feeling" or judgment.
3. Teach the robot to ignore style, not just tell it to ignore style.

In short: The robot is smart, but it's also a bit of a snob. It loves the way it was taught to speak, and it unfairly judges anyone who speaks differently.

Technical Summary

1. Problem Statement

As Large Language Models (LLMs) are increasingly deployed as automated graders in educational settings, there is a critical concern regarding fairness and implicit bias. While LLMs promise scalability and efficiency, they operate on the untested assumption that they can evaluate content correctness independently of a student's linguistic background, writing style, or cultural context.

The core problem addressed is whether LLMs penalize students for surface-level stylistic variations (e.g., informal language, non-native phrasing, or minor grammar errors) even when the underlying conceptual content is correct. If models penalize these variations, they risk systematically disadvantaging non-native speakers, first-generation students, and those from informal educational backgrounds, thereby reinforcing educational inequalities rather than democratizing access.

2. Methodology

The study employed a controlled experimental design to isolate writing style from content correctness.

• Dataset Construction:

  • Scope: 180 student responses across three subjects: Mathematics (20 questions), Programming (20 Python tasks), and Essay/Writing (20 argumentative prompts).
  • Perturbation Strategy: For each of the 60 base questions, three distinct surface-level perturbations were applied while strictly preserving content correctness:
    1. Grammar Errors: Spelling mistakes, punctuation errors, and grammatical inconsistencies.
    2. Informal Language: Conversion to casual, conversational style (slang, contractions).
    3. Non-native Phrasing: Patterns typical of non-native speakers (article misuse, direct translation artifacts).
  • Ground Truth: Human experts assigned scores (1–10 scale) based solely on content correctness, completeness, and reasoning depth.

• Models Evaluated:

  • LLaMA 3.3 70B (Meta): English-dominant training corpus, Western-centric.
  • Qwen 2.5 72B (Alibaba): Multilingual training corpus with significant representation of non-English languages.
  • Rationale: Comparing two state-of-the-art open-source models with similar parameter counts but divergent training data origins to distinguish model-specific artifacts from systemic biases.

• Grading Protocol:

  • Models were prompted with a standardized rubric (1–10 scale) and explicit anti-bias instructions: "Do NOT penalize for grammar, spelling, punctuation, or writing style" and "Do NOT penalize for informal language or non-standard English."
  • Temperature was set near zero (0.0–0.01) to ensure reproducibility.

• Statistical Analysis:

  • Bias was measured as the score delta ($\Delta = \text{Base Score} - \text{Perturbed Score}$).
  • Metrics included: Paired t-tests ($p < 0.05$), Cohen's $d$ for effect size, Pearson correlation with human scores, and Mean Absolute Error (MAE).

3. Key Contributions

1. Controlled Perturbation Framework: Developed a rigorous method to decouple writing style from content correctness, enabling the direct measurement of surface-level bias.
2. Subject-Dependent Bias Discovery: Revealed a sharp contrast between objective (STEM) and subjective (Essay) grading, demonstrating that bias is not uniform across domains.
3. Prompt Engineering Limitations: Demonstrated that explicit instructions to ignore style are insufficient to prevent bias, challenging the efficacy of prompt-based debiasing strategies for high-stakes applications.

4. Key Results

A. The "Subjectivity Gradient"

The most significant finding is that bias magnitude correlates strongly with the subjectivity of the task:

• Essay/Writing: Exhibited severe, statistically significant bias ($p < 0.05$) across all perturbation types for both models.
  • Informal Language: Received the heaviest penalty. LLaMA deducted an average of 1.90 points; Qwen deducted 1.20 points.
  • Non-native Phrasing: LLaMA deducted 1.35 points; Qwen deducted 0.90 points.
  • Effect Sizes: Ranged from medium ($d=0.64$) to very large ($d=4.25$). A $d=4.25$ is exceptionally high in behavioral research, representing a near-two-point drop on a 10-point scale (comparable to the difference between a B+ and C+).
• Mathematics & Programming: Showed minimal to zero bias. Most conditions failed to reach statistical significance.
  • Programming tasks showed virtually no penalty ($\Delta \approx 0.00$) because outputs are objectively verifiable.
  • Mathematics showed slight bias only for informal language, but generally remained fair.

B. Model Comparison

• LLaMA 3.3 70B: Exhibited a higher magnitude of bias (Overall Bias Index: 0.472) but was significant in fewer conditions (33.3%).
• Qwen 2.5 72B: Exhibited a lower magnitude of bias (Overall Bias Index: 0.350) but was more pervasive, showing significant bias in 44.4% of conditions.
• Convergence: Despite different training data (Western vs. Multilingual), both models followed the same bias hierarchy: Essay > Math > Programming and Informal > Non-native > Grammar.

C. Failure of Prompt-Based Debiasing

Despite explicit instructions to ignore style, both models penalized non-standard writing. This suggests the bias is encoded deeply within the model's weights (representational level) and cannot be overridden by surface-level prompt engineering.

D. Human-LLM Agreement

Correlation between LLM scores and human ground truth was weak overall ($r \approx 0.31–0.34$). Agreement was strongest for Programming and weakest for Mathematics (likely due to ceiling effects where models gave perfect scores regardless of style).

5. Significance and Implications

• Educational Equity: The study provides empirical evidence that automated grading systems could systematically lower the grades of non-native speakers and students from informal backgrounds, not due to lack of knowledge, but due to stylistic mismatches with the model's training data.
• Deployment Risks: Institutions relying on LLMs for grading subjective tasks (essays, open-ended responses) risk automating discrimination. The "subjectivity gradient" suggests that LLMs are currently safe for objective STEM grading but dangerous for humanities and social science assessment.
• Technical Limitations: The findings challenge the assumption that "careful prompting" is a sufficient mitigation strategy. The paper argues for more fundamental interventions, such as:
  • Style-aware fine-tuning on diverse datasets.
  • Architectural modifications to decouple content from style.
  • Mandatory bias auditing protocols using perturbation-based testing before institutional adoption.
• Moral Imperative: As LLMs proliferate in education, ensuring they serve all students equitably is framed not just as a technical challenge, but a moral necessity to prevent the reinforcement of existing socioeconomic and linguistic inequalities.