Technical Summary: SUBDATA – Bridging Heterogeneous Datasets for Theory-Driven Evaluation of LLM Perspective Alignment

1. Problem Statement

As Large Language Models (LLMs) become increasingly capable of capturing nuanced human perspectives, researchers are exploring their utility in subjective tasks such as hate speech detection. A critical challenge in this domain is perspective alignment: the ability of models to reflect diverse human viewpoints (e.g., political or demographic) across different contexts.

Current evaluation efforts are hindered by a lack of standardized resources. Studies often rely on inconsistent datasets with varying annotation schemes and demographic categorizations, making it difficult to compare how differently aligned models perform on downstream tasks. Furthermore, existing evaluation methodologies often struggle with the inherent subjectivity of "ground truth" in tasks like hate speech detection, where human annotations naturally diverge across ideological lines. There is a need for a framework that enables systematic, theory-driven evaluation of how perspective conditioning influences model behavior without relying on contested objective labels.

2. Methodology

The paper proposes a two-step framework to address these challenges: Dataset Standardization via the SUBDATA library, and Theory-Driven Hypothesis Testing.

2.1 SUBDATA Library Construction

SUBDATA is an open-source Python library designed to collect, harmonize, and standardize heterogeneous datasets for subjective tasks.

• Data Collection: The authors employed a multi-phase approach to identify datasets containing explicit annotations of targeted demographic groups. This involved leveraging domain expertise, examining existing repositories (e.g., hatespeechdata.com), and conducting systematic literature searches.
• Selection Criteria: Datasets were selected based on their focus on subjective constructs (hate speech, toxicity) and the presence of granular target group annotations (e.g., specifying "Jews" or "women" rather than generic "other" groups).
• Unified Taxonomy: To resolve inconsistencies in labeling across ten selected datasets (totaling ~852,000 instances), SUBDATA implements a standardized taxonomy. This taxonomy maps diverse original keywords (e.g., "Kikes," "Jewish people," "blacks") into consistent target categories across nine dimensions: age, disability, gender, migration, origin, political, race, religion, and sexuality.
• Processing: The library handles data by splitting multi-target annotations into separate instances, removing duplicates, and excluding entries without specific target groups. It does not host the raw datasets but provides the mapping logic to unify them.

2.2 Theory-Driven Evaluation Framework

Building on standardized data, the framework proposes a workflow to test hypotheses derived from social or political theory:

1. Theory (T): Identify established theories predicting differences in perception across demographic or ideological groups.
2. Hypothesis (H): Formulate testable hypotheses regarding how LLMs aligned with specific perspectives will classify content.
3. Experiment (E): Conduct controlled experiments using the standardized datasets to measure classification differences between differently aligned models.

This approach avoids the need for consensus on "ground truth" labels. Instead, it measures the divergence in model outputs conditioned on different perspectives, providing a direct lens on how perspective conditioning shapes downstream behavior.

3. Key Contributions

1. SUBDATA Library: An open-source tool that standardizes heterogeneous hate speech datasets, unifying annotation schemes and demographic taxonomies to enable consistent cross-dataset analysis.
2. Unified Taxonomy: A structured mapping system that resolves inconsistencies in target group labeling (e.g., mapping "mexicans" to either race or origin categories based on context) and introduces "unspecified" suffixes for generic labels.
3. Evaluation Framework: A novel, theory-driven methodology for evaluating LLM perspective alignment that relies on comparative model behavior rather than adherence to subjective ground truth.
4. Empirical Demonstration: A concrete use case applying the framework to test how persona-conditioned LLMs (aligned with left- vs. right-leaning political stances) detect hate speech targeting specific demographics.

4. Experimental Results

The authors applied the framework to three open-source instruction-tuned LLMs (Mistral-7B, Llama-3.1-8B, Qwen2.5-7B) using 20 left-leaning and 20 right-leaning personas per model.

• Asymmetric Sensitivity: Across all models and target groups, left-oriented personas consistently yielded higher hate speech detection rates than right-oriented personas. This effect was observed for both minority groups (e.g., Blacks, Muslims, LGBTQ+) and majority groups (e.g., Whites, Christians, Men).
• General Tightening vs. Selective Protection: The results suggest that left-leaning conditioning leads to a general tightening of classification thresholds rather than selective protection of specific groups. This contradicts the hypothesis that right-leaning personas would be more protective of majority groups.
• Model Variance:
  • Mistral showed the strongest divergence between left and right personas (Odds Ratio = 1.364).
  • Llama exhibited the smallest persona effect (OR = 1.068) but the highest baseline detection rates overall.
  • Qwen fell in between regarding persona sensitivity but had the lowest baseline detection.
• Specific Anomalies: Llama showed unusually high protection for Whites (0.656 detection rate under left personas), potentially reflecting its U.S.-centric training distribution where race discourse often centers on contrasts involving Whites.

5. Significance and Claims

The paper positions SUBDATA as a foundational resource for the emerging field of LLM perspective alignment. Its significance lies in:

• Enabling Rigorous Comparison: By standardizing datasets, it allows researchers to move beyond inconsistent, study-specific resources to make cumulative, comparable claims about model behavior.
• Addressing Subjectivity: The theory-driven framework acknowledges the inherent subjectivity of tasks like hate speech detection. It shifts the evaluation metric from "accuracy against a contested ground truth" to "quantifiable differences in perspective-conditioned behavior."
• Community Building: The authors invite the research community to contribute additional datasets and extend the library beyond hate speech to other subjective constructs (e.g., misinformation), aiming to evolve SUBDATA into a comprehensive multi-construct benchmark suite.

The authors maintain a modest stance, acknowledging that the current taxonomy involves manual mapping choices and that the library aggregates existing annotations without re-labeling. They emphasize that the tool is designed to improve evaluation transparency and understanding, not to enable the creation of harmful or biased systems.