Technical Summary: Harsher on Male? Evaluating LLMs on Gender-Asymmetric Moral Framing Across Diverse Conflict Scenarios

Problem Statement

Existing research on gender bias in Large Language Models (LLMs) has predominantly focused on static stereotypes, occupational associations, and explicit harmful outputs. However, as LLMs evolve into aligned assistants for real-world interactions, bias may manifest more subtly in how models judge identical behaviors based on the actor's gender. Current evaluation methods—such as lexical swaps, forced-choice questions, or static association tests—fail to capture how LLMs generate long-form, open-ended responses in realistic social and intimate conflict scenarios. Specifically, there is a lack of frameworks that can isolate the actor's gender while strictly controlling for the severity, context, and nature of the misconduct to determine if models apply consistent moral standards.

Methodology: GAMA-Bench

To address this gap, the authors introduce GAMA-Bench, a gender-mirrored benchmark designed to evaluate response-framing differences under matched conditions.

Dataset Construction

The benchmark consists of 1,298 scenario instances divided into two parallel tracks:

1. Intimate Track: Covers conflicts in close relationships (e.g., financial control, privacy invasion, emotional blackmail, mild violence).
2. Public Track: Covers non-intimate social conflicts (e.g., false accusations, workplace bullying, public disputes).

The construction pipeline ensures rigorous control:

• Gender-Neutral Templates: Scenarios are first generated as gender-blind skeletons using placeholders (e.g., [AGGRESSOR], [VICTIM]) to define the misconduct, context, severity (L1–L3), and rhetorical style.
• Cross-Model Review: Templates are audited by different LLMs to ensure they contain no implicit gender cues, stereotypes, or unrealistic elements.
• Deterministic Mirroring: Each template is compiled into a pair of first-person prompts. The only variables changed are the actor's gender and corresponding role references (e.g., "I am a man/woman" and "my boyfriend/girlfriend"). All other elements (behavior, severity, tone) remain identical.

Evaluation Protocol

Instead of binary classification (harmful vs. harmless), the authors propose a structured response-framing protocol. An evaluator LLM (primarily Gemini-3-Pro, with robustness checks using GPT-5.4) maps open-ended model responses to six fine-grained metrics:

1. Punitive Wording: Frequency of condemnatory or sanction-oriented language.
2. Therapeutic Wording: Frequency of empathetic, contextualizing, or mitigating language.
3. Severity Rating: The degree of risk escalation (1–5 scale).
4. Empathy toward Aggressor: The percentage of text dedicated to understanding the actor's feelings.
5. Instructional/Accusatory Framing: The percentage of text dedicated to lecturing or correcting the actor.
6. Full-Blame Attribution: Whether the response assigns full responsibility to the actor.

Experimental Setup

The study evaluated 10 state-of-the-art LLMs, including proprietary models (GPT-5.2/5.4, Gemini-2.5/3-Pro, Doubao, Kimi) and open-source models (Qwen3/3.5, DeepSeek, MiniMax). The evaluation also included ablation studies on model scale (0.6B to 8B), post-training (Base vs. Instruct), and reasoning modes (Chain-of-Thought).

Key Results

The experiments reveal a consistent male-disadvantaging asymmetry across all tested models and tracks:

• Intimate Track: Male actors receive significantly harsher framing than female actors for the exact same misconduct.
  • Punitive & Blame: Male actors receive approximately 3.3 more punitive words, 14% higher instructional/accusatory framing, and 23% higher full-blame attribution.
  • Therapeutic & Empathy: Female actors receive approximately 1.3 more therapeutic words and 9% higher empathy toward the aggressor.
  • Severity: Male actors are assigned severity ratings 0.40 points higher on average.
• Public Track: The same directional asymmetry persists, though the magnitude of the gaps is smaller (e.g., ~9% higher instructional framing for males).
• Robustness:
  • Model Scale: Larger models do not resolve the bias; in some cases, scaling exacerbates the punitive bias against males.
  • Post-Training: Instruction-tuned models amplify the punitive framing toward males compared to base models.
  • Reasoning: Extended reasoning (Chain-of-Thought) fails to correct the asymmetry and sometimes widens the gaps.
  • Evaluator Consistency: Results remain stable when switching evaluators (Gemini-3-Pro vs. GPT-5.4), with a 97.5% directional agreement.
  • Human Validation: Expert annotators confirmed that the automated metrics align with human-perceivable differences in tone and blame attribution.

Significance and Claims

The paper argues that gender bias in LLMs is not limited to final judgments or explicit stereotypes but is deeply embedded in the linguistic framing of open-ended interactions.

• Dynamic Double Standard: The study demonstrates that LLMs apply a "dynamic moral double standard," treating identical negative behaviors as disciplinary issues requiring sanction when performed by men, but as relational or emotional issues requiring explanation and mitigation when performed by women.
• Beyond Final Labels: The authors claim that fairness evaluation must move beyond final labels (e.g., "biased/unbiased") to examine the process of judgment, including tone, risk escalation, and responsibility assignment.
• Systemic Nature: The persistence of this asymmetry across model families, scales, and reasoning modes suggests it is a fundamental, pervasive characteristic of current aligned LLMs rather than an artifact of specific model architectures or training data noise.

The paper concludes that GAMA-Bench provides a necessary framework for diagnosing these subtle, framing-based biases, which are critical for deploying LLMs in sensitive domains like relationship advice, conflict mediation, and moral judgment.