Gendered Prompting and LLM Code Review: How Gender Cues in the Prompt Shape Code Quality and Evaluation

1. Problem Statement

The integration of Large Language Models (LLMs) into software engineering workflows has democratized coding but introduced new vectors for bias. While existing research focuses on bias in training data or output content, this paper addresses a critical, under-explored gap: how gendered linguistic styles in user prompts influence LLM code generation and, more significantly, automated code review.

The authors investigate three core Research Questions (RQs):

• RQ1: Is user gender predictable from the linguistic style of coding prompts?
• RQ2: Does gendered prompting lead to differences in the functional correctness or static quality of generated code?
• RQ3: Do LLMs acting as automated code reviewers exhibit bias based on the gender cues in the prompt or the perceived persona of the author?

2. Methodology

The study employs a mixed-methods approach across three complementary studies to isolate variables ranging from real-world usage to controlled simulations.

Study 1: Real-World Prompt Analysis

• Data: 753 prompts from 30 participants (15 male, 13 female, 2 non-binary) collected via survey, covering real-world Python coding tasks.
• Analysis:
  • Linguistic Profiling: Analyzed prompt length, pronoun usage, hedging (modal verbs), and "involved-informational" ratios (based on Biber's framework).
  • Gender Prediction: Fine-tuned a RoBERTa model to predict user gender from prompt text alone.
  • Code Quality: Evaluated generated code using static analyzers (Pylint, Radon) and user satisfaction metrics.
• Goal: To characterize natural linguistic differences and their correlation with code quality.

Study 2: Controlled User Experiment

• Design: 59 participants (73.5% male, 26.5% female) solved two standardized coding tasks (password checker, hashtag validator) using their LLM of choice.
• Evaluation:
  • Correctness: Unit tests.
  • Quality: Cyclomatic complexity, maintainability index, and Pylint scores.
  • Review: Solutions were submitted to LLM-based reviewers (simulated "Code Reviewers") to measure approval rates.
• Goal: To determine if human gender affects code outcomes when using LLMs and if LLM reviewers treat male/female-authored code differently.

Study 3: Synthetic Simulation (LLM-Only)

• Design: Removed human variation by using synthetic prompts with systematically varied gender cues.
  • Prompt Variants: Male-coded, Female-coded, Neutral, and Self-introduction variants (using names "Jack" and "Sarah" and stereotypical adjectives).
  • Tasks: Five algorithmic programming tasks (e.g., Roman numeral conversion, spiral matrix).
  • Reviewers: LLMs with system messages defining male, female, or neutral personas.
• Goal: To causally isolate the effect of prompt style and reviewer persona on code generation and evaluation, controlling for user skill.

3. Key Contributions

1. Linguistic Characterization: Empirical evidence that female-authored prompts in coding contexts exhibit more indirect, involved, and hedged language (e.g., "Can you...", "I need...") compared to the more direct, imperative style of male-authored prompts.
2. Decoupling Generation from Evaluation: The study demonstrates that while gendered prompting styles do not significantly impact the functional correctness of generated code, they do significantly impact how LLMs evaluate that code.
3. Identification of "LLM-as-Judge" Bias: The primary contribution is the discovery that LLMs acting as automated reviewers exhibit systematic biases, approving female-authored code more frequently than male-authored code despite comparable quality.
4. Impact on Code Structure: Gendered prompt styles causally influence code structure; female-coded prompts tend to generate longer, more maintainable code, while male-coded prompts generate more compact code.

4. Key Results

Prompt Linguistics (Study 1 & 2)

• Linguistic Differences: Women used significantly more personal pronouns, pragmatic markers, and hedges (e.g., "can you," "maybe"). Men used more direct commands.
• Predictability: Gender prediction from prompts alone was moderate (RoBERTa F1 $\approx$ 0.60), suggesting that while linguistic traces exist, they are subtle and confounded by experience levels.
• Code Correctness: Across all studies, no significant difference was found in functional correctness (unit test pass rates) between code generated by men, women, or via gendered prompts.

Code Quality Metrics (Study 1, 2, & 3)

• Static Analysis: No significant differences in Pylint scores or cyclomatic complexity based on user gender in human studies.
• Structural Impact (Study 3): In the controlled simulation, female-coded prompts consistently produced:
  • Longer code (higher Lines of Code).
  • Higher Maintainability Index.
  • Lower Cyclomatic Complexity.
  • Conversely, male-coded prompts produced more compact code with higher global Pylint scores.

LLM Reviewer Bias (Study 2 & 3)

• The Approval Gap: In Study 2, LLM reviewers approved 70.6% of female-authored submissions vs. 62.9% of male-authored submissions ($p \approx 0.008$), despite identical correctness.
• Model Variance: The bias varied by provider. Groq (LLaMA models) showed the most significant bias, while Anthropic showed the least.
• Persona Interaction: In Study 3, reviewer personas (Male/Female/Neutral) reacted differently to prompt styles. For example, female reviewers were stricter for Groq and OpenAI, while male reviewers showed sensitivity to male-coded prompts in Deepseek.

5. Significance and Implications

• Shift in Bias Focus: The paper argues that fairness risks in LLM-assisted programming are shifting from generative accuracy (which appears robust across genders) to evaluation bias. As LLMs are increasingly deployed as automated code reviewers, graders, and hiring filters, they may systematically favor or penalize specific communication styles.
• The "LLM-as-Judge" Risk: The findings suggest that simply stripping demographic attributes (names/genders) from prompts is insufficient. Gendered information is encoded in stylistic features (politeness, indirectness) that LLMs interpret, leading to biased outcomes in automated review pipelines.
• Socio-Technical Governance: The authors call for a socio-technical perspective. Technical debiasing must be coupled with governance regarding when and how LLMs are used as judges. Educational and professional tools should support diverse prompting styles rather than enforcing a single "expert" (often male-coded) communication norm.
• Limitations: The study acknowledges small sample sizes, a binary view of gender (excluding non-binary data due to low counts), and the use of short coding tasks rather than full software projects. Future work requires larger, more diverse datasets and human-in-the-loop validation.

Conclusion: The paper concludes that while LLMs can equalize coding outcomes for different genders, the automated evaluation layer introduces a new form of bias where "female-coded" communication styles receive preferential treatment in approval rates, highlighting a critical need to audit LLM-based review systems for demographic and stylistic bias.