Bye Bye Perspective API: Lessons for Measurement Infrastructure in NLP, CSS and LLM Evaluation

1. Problem Statement

The paper addresses the impending closure of the PERSPECTIVE API (scheduled for late 2026), which has served as the de facto standard for automated toxicity measurement in Natural Language Processing (NLP), Computational Social Science (CSS), and Large Language Model (LLM) evaluation.

The authors argue that the research community's reliance on this single, proprietary tool has created a structural dependence that led to significant epistemic and methodological failures:

• Black-Box Nature: The model was updated silently without versioning, changelogs, or disclosure, making results irreproducible over time.
• Construct Validity Issues: The tool operationalized "toxicity" based on a contested behavioral assumption (that toxicity causes disengagement) which empirical evidence contradicts. It also failed to distinguish clearly between toxicity, hate speech, and other attributes.
• Circular Evaluation: The API was used not just as a measurement tool, but as a source of training data and a ground-truth standard for evaluating other models. This created a self-reinforcing loop where models were optimized to match the API's specific biases rather than actual human judgment of harm.
• Context Stripping: The API processes isolated sentences, stripping away conversational context, speaker intent, and target identity, leading to systematic misclassification of reclaimed slurs, counterspeech, and nuanced political discourse.

2. Methodology

The authors employ a critical infrastructure analysis and measurement validation framework adapted from social science methodologies (specifically construct validation). The methodology involves:

• Historical & Usage Analysis: Tracing the adoption of PERSPECTIVE API across NLP, CSS, and LLM evaluation research to identify how it became a standard (path dependence and network effects).
• Validation Pipeline Critique: Comparing the API's development against standard measurement validation procedures:
  1. Systematisation: Analyzing the conceptual definition of toxicity (the "construct") and its behavioral assumptions.
  2. Operationalisation: Examining how the concept was translated into a technical instrument (input representation, model architecture, and output scoring).
  3. Provision: Investigating the transparency of the model's updates, training data, and annotation processes.
• Empirical Evidence Review: Synthesizing findings from prior studies (e.g., Gervais et al., 2025; Pozzobon et al., 2023) that demonstrated the API's poor performance in specific contexts (e.g., low F1 scores on political incivility, inversion of LLM benchmarks upon model retraining).
• Structural Analysis: Identifying the "structural circularity" where the tool defines the standard it is supposed to measure, laundering social biases into technical neutrality.

3. Key Contributions

The paper makes three primary contributions to the field:

1. Diagnosis of Measurement Failure: It formally documents that the field built toxicity research on a tool that was never validated for its central role. It highlights that the API's "scores" were uncalibrated probabilities relative to an artificial baseline, not interpretable measures of real-world harm prevalence.
2. Identification of Structural Circularities: The authors demonstrate how the API's dual role as a data generator and an evaluation standard created a feedback loop. This resulted in models that were "good" at mimicking the API's specific biases (e.g., over-moderating identity-laden content) rather than being robust detectors of harm.
3. A Framework for Future Infrastructure: The paper proposes 10 specific requirements (5 Technical, 5 Governance) for a successor measurement infrastructure to prevent a recurrence of these failures.

4. Key Results & Findings

• Invalidity of the Construct: The API's definition of toxicity (rude comments causing disengagement) is empirically false; toxic content often increases engagement. Consequently, the tool measures "disagreement" or "offensiveness" rather than the intended construct of "harmful toxicity."
• Reproducibility Crisis: Due to silent model retraining, 49% of previously toxic prompts were reclassified differently upon rescore. This renders historical benchmarks (like RealToxicityPrompts and HELM) irreproducible and incomparable over time.
• Systematic Bias: The API exhibits consistent over-moderation of content related to marginalized groups (e.g., LGBTQIA+, reclaimed slurs) and under-detection of implicit hate speech, driven by its reliance on surface features and lack of context.
• The "Moving Target" Problem: Without versioning, researchers could not determine if they were querying the same instrument, making longitudinal studies and meta-analyses scientifically invalid.
• Risk of Replacement: The authors warn that replacing PERSPECTIVE with closed-source LLM endpoints (e.g., OpenAI Moderation) would replicate these failures, as these tools lack transparency, versioning, and community governance.

5. Significance and Recommendations

The paper argues that the closure of PERSPECTIVE API is not merely a tooling issue but a measurement crisis that necessitates a fundamental shift in how the community approaches evaluation infrastructure.

Proposed Requirements for Successor Infrastructure:

Immediate Actions Recommended:

• Archiving: Researchers must immediately archive PERSPECTIVE scores for all major benchmarks with full method documentation (query dates, thresholds, versions).
• Shift to Open-Weight Models: New projects should prefer open-weight models with explicit versioning over closed endpoints.
• Epistemic Reflexivity: The community must treat the construction of measurement infrastructure as a core research contribution requiring the same rigor as model development, moving away from treating it as "service labor."

Conclusion:
The paper concludes that the NLP and CSS communities have the technical capacity to build valid, reliable, and fair measurement infrastructure. The barrier is not technical but sociological: a collective decision to prioritize infrastructure building as scientifically valuable work rather than outsourcing it to proprietary, opaque vendors.