Technical Summary: Annotator Positionality as Signal: Psychometric Weighting for Anti-Autistic Ableism Detection

Problem Statement

Large language models (LLMs) are increasingly deployed in decision-making pipelines, such as content moderation, where they risk amplifying or suppressing community perspectives. While prior research has identified general disability-related biases in LLMs, the specific mechanisms by which these models conceptualize and detect anti-autistic ableism remain underexplored. A critical challenge in this domain is the insider/outsider distinction: terms reclaimed by the autistic community (e.g., "aspie," "autie") to signal solidarity may be misclassified as ableist by automated systems or human annotators lacking community-grounded knowledge. Furthermore, standard evaluation methods relying on majority-vote aggregation systematically underweight autistic and autism-accepting perspectives, treating annotator identity as noise rather than a meaningful signal. This leads to ground truths that fail to capture the contextual nuances of speaker identity, reclaimed language, and community impact.

Methodology

The authors propose a bias-aware evaluation framework that integrates psychometric weighting with controlled LLM evaluations.

1. Psychometrically Weighted Ground Truth

Instead of simple majority voting, the study constructs a ground truth by weighting human annotator labels based on their measured bias levels and autistic traits.

• Data Source: 2,120 sentences from the AUTALIC dataset, labeled as ableist (1) or not (0).
• Annotator Characterization: Annotators completed three psychometric instruments:
  • SATA (Societal Attitudes Toward Autism): Measures explicit acceptance.
  • AQ (Autism-Spectrum Quotient): Quantifies autistic traits.
  • IAT (Implicit Association Test): Measures implicit biases toward autism.
• Weighting Scheme: A composite reliability score ($R_i$) is calculated by min-max normalizing and averaging the three instrument scores (with IAT inverted so higher values indicate lower bias). These scores are normalized relative to the team mean to produce a final weight ($W_i$). The ground truth label ($\hat{y}$) is the weighted mean of annotator labels.
• Validation: This composite weighting was shown to produce a significantly stricter ground truth than majority voting ($p < .001$), whereas AQ-only weighting did not ($p = .651$).

2. LLM Evaluation Setup

• Models: Twelve publicly available LLMs were evaluated, ranging from 135M to 20B parameters, including instruction-tuned, reasoning-capable, safety-oriented, and uncensored variants.
• Prompting Conditions: Eight models were tested under seven distinct conditions:
  1. Zero-Shot (ZS): Definition-based framing.
  2. Chain-of-Thought (CoT): Three-step reasoning (speaker identity, keyword vs. meta-context, harm assessment).
  3. In-Context Learning (ICL): Two sub-conditions using examples from high-AQ (autistic-proximate) and low-AQ (non-autistic) annotators.
  4. Persona Prompts: Three sub-conditions varying identity language (Identity-First, Person-First, or neutral).
• Metrics: Primary agreement was measured using Cohen's $\kappa$ against the weighted ground truth. Secondary metrics included F1, precision, recall, and accuracy.

3. Psychometric Evaluation of LLMs

To assess latent biases, LLMs were administered the SATA and AQ instruments. To mitigate "recognition-based responding" (where models mimic socially desirable answers due to training data familiarity), the study employed a masked rewrite of the instruments. This involved item randomization, phrasing variation, and the inclusion of filler and validity (dummy) items to obscure the questionnaire's purpose.

4. Error Analysis

A triangulated error analysis was conducted on 92 sampled LLM outputs. Three analysts used distinct lenses: (1) thematic content, (2) disability-studies linguistic features, and (3) reasoning structure.

Key Results

1. Ground Truth Construction is Not Neutral

• Stricter Baseline: Psychometrically weighted $\kappa$ was consistently lower than unweighted (majority-vote) $\kappa$ across 56 model-condition combinations (mean $\delta\kappa = -0.012$). This confirms that majority voting systematically underrepresents autistic perspectives.
• Alignment Gaps: LLMs showed significantly higher agreement with "reliable" annotators (high trust scores) than with "autistic-proximate" annotators, validating the necessity of the composite weighting scheme.
• Model Variance: Safety-oriented classifiers (LLaMA Guard 3, GPT-OSS) outperformed general-purpose models, suggesting task-specific safety training encodes sensitivity to disability harm that general prompting cannot recover.

2. Classification Performance

• Overall Low Performance: Mean weighted $\kappa$ was 0.118 and mean F1 was 0.237, indicating near-chance performance on a high-stakes task.
• Precision-Recall Imbalance: Models exhibited systematic over-classification (high recall, low precision), flagging community-reclaimed terms as ableist.
• Prompting Limitations:
  • CoT: While Chain-of-Thought eliminated "topic deflection" (justifying non-ableist labels by claiming the text isn't about autism), it failed to significantly improve "positionality reasoning" (understanding speaker identity).
  • ICL & Personas: In-context learning and persona prompts did not consistently improve performance; in some cases, they destabilized high-performing models.

3. Reasoning Failures

The error analysis identified four primary failure modes:

1. Surface-Form Reasoning: Models relied heavily on keyword matching (41% of excerpts) rather than context. They frequently mislabeled reclaimed terms (e.g., "aspie") as ableist and missed deficit-based framings lacking negative keywords.
2. Positionality Blindness: Explicit reasoning about speaker identity was rare (10% overall, 13% under CoT) and absent in zero-shot conditions.
3. Topic Deflection: Models often justified non-ableist labels by claiming the text did not concern autism, a behavior eliminated by CoT but replaced by other errors.
4. Hallucination & Stereotypes: Models invoked stereotypes and, in some cases, hallucinated topics (e.g., misidentifying a sentence about Autism Speaks as anti-Semitism).

4. Latent Biases in LLMs

• Masking Reveals Bias: When instrument-recognition cues were masked, LLMs showed a significant divergence in responses: SATA scores declined (indicating lower acceptance) and AQ scores increased (indicating higher autistic traits in the clinical range) compared to the original instruments.
• Interpretation: This suggests that standard self-reports by LLMs elicit socially desirable responses that conceal latent anti-autistic attitudes.

Significance and Claims

The paper claims to provide a replicable template for community-grounded, bias-aware evaluation in harmful language detection. Its primary contributions are:

1. Methodological: Demonstrating that psychometric weighting creates a stricter, more accurate ground truth than majority voting for detecting anti-autistic ableism, specifically by up-weighting annotators with high autistic traits and low implicit bias.
2. Empirical: Revealing that current LLMs, even with advanced prompting strategies like CoT and personas, fail to reason beyond surface-level keywords regarding speaker identity and reclaimed language.
3. Diagnostic: Showing that masked psychometric evaluations expose latent anti-autistic biases in LLMs that are hidden by standard self-reporting mechanisms.

The authors conclude that task-specific fine-tuning on community-labeled data is a precondition for deployment-grade performance in this domain, rather than an optional enhancement. They emphasize that current models operate with near-chance accuracy on tasks affecting autistic communities, highlighting the urgent need for evaluation frameworks that center community perspectives and positionality.