Technical Summary: Can Data Work be Reparative?

Problem Statement

Contemporary AI systems rely heavily on human "data work" for training, fine-tuning, and evaluation. However, dominant modes of data production are characterized by routine, repetitive labor that treats workers as interchangeable actors, often leading to "cascading issues" in datasets and reinforcing extractive labor practices. Current approaches to "Responsible AI" and safety evaluations often fail to address the foundational power imbalances and epistemic injustices inherent in dataset production, particularly regarding marginalized communities and non-English languages. There is a critical gap in understanding how data work can be reoriented from a site of exploitation to one of repair and redress, specifically addressing the tension between just reward, collective governance, and the recognition of lived experience as expert knowledge.

Methodology

The authors conducted an ethnographic study of Tattle Civic Tech, a civic-tech initiative based in India that builds datasets for training and benchmarking online safety systems from a feminist perspective. The research was conducted by the lead author between April and December 2024 through virtual fieldwork (Slack, online meetings, collaborative tools) and active participation.

The study focused on two specific dataset projects:

1. A Slurs Lexicon Dataset: An expansion of a list of gendered abusive slurs in four Indian languages (Hindi, Tamil, Malayalam, and Indian English), involving 8–10 contributors per session in collaborative annotation sessions.
2. An LLM Safety Benchmark Dataset: A Hindi benchmark for Large Language Models (LLMs) focusing on "hate" and "sex-related crimes" (later reframed as "sexual violence"), funded by ML Commons. This involved 23 contributors (social workers, activists, journalists, psychologists) participating in six 90-minute workshops.

The research employed an abductive approach to data analysis, iteratively coding field notes, interviews (12 total with contributors and Tattle staff), and project materials. The analysis was framed through the lens of reparative justice (drawing on Margaret Walker's work) and Science and Technology Studies (STS), focusing on themes of care, justice, and participation. The authors maintained an engaged positionality, collaborating with Tattle on compensation principles and governance models, and shared drafts with the organization for feedback.

Key Contributions

The paper makes three primary contributions to critical scholarship on data work and AI ethics:

1. Empirical Insights from the Global South: It provides detailed empirical data on the challenges of developing alternative data work approaches outside of Western, Educated, Industrialized, Rich, and Democratic (WEIRD) contexts, extending Responsible AI scholarship beyond its predominant focus.
2. Entanglement of Epistemic and Labor Justice: It demonstrates that concerns regarding the recognition of knowledge (epistemic justice) and the conditions of labor (labor justice) are deeply intertwined. The study argues that these cannot be addressed in isolation, challenging the separation of "data quality" from "worker conditions."
3. Reparative Justice as an Analytical Frame: It introduces reparative justice as a lens for reorienting data work. This framework shifts the focus from merely fixing technical gaps in datasets to resetting the relations of accountability, placing those most harmed by online neglect and exclusion at the center of the process.

Results and Findings

The study identifies significant tensions and challenges in operationalizing a reparative approach to data work:

• Just Reward and Compensation: Tattle attempted to compensate contributors at rates significantly higher than industry standards (approx. 10x the Indian industry rate) to recognize their expertise and lived experience. However, determining "just reward" remains complex due to the diverse material conditions of contributors (some academics, some freelancers) and the lack of industry precedents for valuing "expert" lived experience over simple annotation. Funding bodies often view such approaches as too expensive, limiting scalability.
• Collective Governance and Stewardship: The paper highlights the difficulty of establishing collective governance for datasets. Contributors, while experts in their domains, often viewed platform licensing and commercial use as outside their expertise or prioritized immediate harm reduction over long-term governance control. The transient nature of contributor involvement (sporadic participation based on funding cycles) complicates sustained collective decision-making.
• Reframing Harm and Safety: Contributors actively reshaped the datasets, moving beyond simple annotation to redefine the scope of hazards (e.g., reframing "sex-related crimes" to "sexual violence" and adding "profession" as a category for slurs). This demonstrated that data work could be a site for conceptual repair, not just data entry.
• The Limits of Participation: While Tattle successfully lowered barriers to participation and fostered collaborative discussion, the study reveals that without addressing the underlying power dynamics of who owns the data and who benefits from it, participatory efforts risk remaining transactional. Contributors often felt that engaging with powerful tech platforms (even for free) was a necessity rather than a choice, highlighting a lack of leverage.

Significance and Claims

The paper argues that the work of repairing data work lies fundamentally in resetting the ties of accountability between AI systems, the datasets they rely on, and the humans involved in their production.

• Beyond Technical Fixes: The authors contend that simply including diverse voices in datasets does not constitute repair if the structural relations of accountability remain unchanged. True repair requires establishing shared standards where those most harmed can hold AI companies accountable.
• Interrupting Norms: A reparative lens demands interrupting prevailing norms of data work to center those most harmed by neglect and oversight, rather than centering the AI or the dataset itself.
• A Vision for Responsibility: The paper posits that while a fully realized reparative model may seem difficult to achieve, the attempt to operationalize it offers a "bold vision for responsibility." It suggests that the struggle to define fair compensation and collective governance is not merely a logistical hurdle but a necessary political act to reorient AI practice toward justice.

The authors conclude that their work does not offer prescriptive policy solutions but rather opens a crucial line of inquiry for "doing data work otherwise," emphasizing that the path to responsible AI requires confronting the foundational questions of how humans relate to the systems they help produce.