Aggregated Individual Reporting for Post-Deployment Evaluation

Technical Summary: Aggregated Individual Reporting for Post-Deployment Evaluation

Problem Statement
The paper addresses a critical gap in the AI evaluation ecosystem: the lack of structured, systematic mechanisms for evaluating AI systems after deployment. While static benchmarks and pre-deployment testing (e.g., red-teaming) are standard, they often fail to capture "unknown unknowns"—subtle, emergent, or context-specific failure modes that only manifest in real-world usage. The authors argue that current post-deployment monitoring relies too heavily on unstructured, viral social media feedback or discrete incident databases, which lack the aggregation necessary to distinguish systemic failures from individual anomalies. Furthermore, existing policy directives (e.g., the EU AI Act) call for public feedback but lack concrete implementation frameworks. The core problem is how to operationalize individual user experiences into actionable, fine-grained system evaluations that can drive accountability and safety improvements.

Methodology: The Aggregated Individual Reporting (AIR) Framework
The authors propose Aggregated Individual Reporting (AIR) as a formal framework to bridge individual experiences and system-level evaluation. The framework is defined by three interacting entities and three core components:

1. Entities:

   • Evaluated System: The specific AI system under review (e.g., a chatbot, loan algorithm, or medical scribe).
   • Affected Population: The group of individuals interacting with the system (users, patients, loan applicants, or their close associates).
   • Mechanism Administrator: The entity collecting and aggregating reports. This can be First-party (the developer, e.g., OpenAI), Second-party (an organizational user, e.g., a hospital), or Third-party (an external body, e.g., government or NGO).

2. Core Components:

   • Individual Reporting: Members of the affected population submit reports detailing specific negative experiences or harms.
   • Aggregation for Evaluation: Reports are not treated as isolated incidents but are aggregated over time to identify patterns, trends, and statistical anomalies. The goal is to describe system behavior in a fine-grained manner.
   • Evaluation-Conditional Action: Downstream actions (e.g., model rollback, policy changes, legal investigation) are triggered only when aggregated data meets specific evaluation conditions (e.g., a statistically significant spike in adverse reports for a specific subgroup).

The paper contrasts AIR with existing approaches:

• Incident Databases (e.g., incidentdatabase.ai): These collect discrete, news-worthy events but lack the granularity to describe model behavior or the aggregation mechanisms to identify subtle trends.
• Flaw Disclosure (e.g., bug bounties, red-teaming): These often focus on resolving individual bugs or pre-deployment testing by specific groups, rather than continuous, population-wide monitoring.
• Crowdsourced Clustering (e.g., Anthropic's Clio): While Clio analyzes chat transcripts, it lacks the intentional, user-driven reporting mechanism that highlights specific harms and the structured aggregation required for accountability.

Key Contributions

1. Formal Definition of AIR: The paper provides the first formal taxonomy for AIR, distinguishing it from existing incident databases and flaw disclosure systems by emphasizing the triad of reporting, aggregation, and conditional action.
2. Identification of "Unknown Unknowns": The authors argue that individual reports are uniquely positioned to surface failure modes that centralized evaluations miss, particularly those rooted in "situated knowledge" (e.g., cultural context, specific domain applications) and user creativity (informal red-teaming).
3. Normative Argument for "Democratic" AI: The paper posits that AIR is a necessary component of "democratic" AI. It reframes AI governance not just as a design problem but as an accountability problem, where the "consent of the governed" requires ongoing mechanisms for the public to revoke consent or raise concerns about deployed systems.
4. Implementation Workflow: The paper outlines concrete design decisions required to build an AIR, categorized into:
   • Organizational: Choosing the administrator type and defining the scope of the affected population.
   • Interaction: Determining report affordances (what data is collected), handling reporting behavior (bias in who reports), and incentivizing participation.
   • Methodological: Selecting aggregation methods (e.g., sequential hypothesis testing, clustering) and defining evaluation triggers (e.g., statistical significance vs. ad-hoc trends).

Results and Evidence
As a position paper, the work does not present new experimental results from a deployed AIR system. Instead, it synthesizes evidence from analogous domains and case studies to validate the framework's necessity and potential:

• Case Studies: The paper analyzes the VAERS (vaccine safety) and CFPB (consumer finance) systems to demonstrate how aggregation transforms individual complaints into actionable safety insights (e.g., identifying myocarditis risks in young men via VAERS).
• AI Contextualization: It uses the ChatGPT sycophancy incident (April 2025) as a motivating example where viral, unstructured feedback led to a rollback, contrasting this with less visible but potentially severe issues (e.g., "spiritual delusions") that failed to trigger action due to a lack of effective aggregation.
• Comparative Analysis: It reviews systems like RegretsReporter (YouTube) and FairFare (rideshare wages) to show how statistical aggregation of crowdsourced data can provide evidence for systemic change, even when individual anecdotes are dismissed.

Significance and Claims
The paper claims that AIR is a "practical path" to realizing the promise of post-deployment evaluation and democratic AI. Its significance lies in:

• Bridging the Gap: It offers a shared language and concrete workflow to translate high-level policy mandates (like the EU AI Act) into operational reality.
• Actionability: By insisting on aggregation, the framework moves beyond "hearing" users to "acting" on their collective experiences, providing the statistical evidence needed to compel organizational change or legal accountability.
• Empirical Imperative: The authors modestly claim that while the framework is theoretically sound, its success is an "empirical question." They call for researchers, activists, and policymakers to co-develop and test AIR implementations, acknowledging that failure modes (e.g., biased reporting, lack of organizational incentive) must be addressed through real-world iteration rather than theoretical perfection.

In summary, the paper argues that individual experiences are an integral, non-redundant data source for AI safety, and that Aggregated Individual Reporting is the necessary mechanism to convert these experiences into the collective knowledge required for effective post-deployment governance.