Technical Summary: ASSERT – A Measurement Pipeline for GenAI Audits

1. Problem Statement

Audits of Generative AI (GenAI) systems frequently summarize system behavior as a single reported rate (e.g., a compliance percentage). Researchers and stakeholders use these rates to compare systems, track regressions, and gate deployments. However, a reported rate is a function of both the system under audit and the specific measurement choices made to produce it (e.g., test case generation, simulated user personas, judging models, and evidentiary standards).

When these measurement choices are left implicit, a change in the reported rate creates ambiguity: it is unclear whether the system's behavior changed or if the measurement instrument changed. Furthermore, differences in reported rates between audits are difficult to interpret, compare, or reproduce without a clear record of the underlying specifications. Existing benchmarks often fix a single static set of test cases to maximize comparability, while generative auditing pipelines dynamically generate inputs but may lack explicit documentation of the measurement framework.

2. Methodology: The ASSERT Pipeline

The authors introduce ASSERT (Adaptive Spec-driven Scoring for Evaluation and Regression Testing), a specification-driven measurement pipeline designed to make the audit process transparent and reproducible. The pipeline operationalizes the measurement framework proposed by Chouldechova et al. (2024), moving from a broad "background concept" to a "systematized concept" and then to concrete instruments.

The pipeline consists of three core components:

2.1 The Measurement Task

Researchers define the audit by specifying four elements:

1. Systematized Concept: A narrow, explicit definition of the behavior being measured (e.g., "conversational misrepresentation" rather than the broad concept of "deception"). This includes definitions, observable behavioral patterns, and permissibility criteria.
2. Instance Type: The unit of analysis (e.g., a multi-turn transcript).
3. Population of Interest: The distribution of instances the measurement intends to describe, determined by the test design and rollout protocol.
4. Estimand: The target quantity (e.g., mean compliance over the population).

In ASSERT, researchers provide a plain-language behavior description and application context. An LLM assists in drafting observable patterns and marking them as problematic or acceptable, which are then reviewed and edited by researchers before execution.

2.2 The Measurement Instruments

The pipeline translates the task specification into concrete instruments:

• Behavioral Rubric: Converts the concept specification into scoring instructions for a judge, defining the evidentiary standard (what evidence is required to assign a label).
• Test-Case Generation: Uses an LLM to instantiate a declared test design (crossing behaviors with context dimensions like interaction conditions or user personas) into concrete prompts or scenario briefs.
• Rollout Protocol: Executes test cases against the evaluated GenAI system. For multi-turn audits, a simulated user (an LLM) drives the interaction.
• Judge: An LLM applies the rubric to the resulting transcripts, assigning binary compliance labels and rationales.
• Estimator: Aggregates labels into a reported rate (the unweighted proportion of compliant transcripts).

2.3 The Measurement Specification

A critical component of ASSERT is the written specification (e.g., a spec.yaml file) that records every choice made in the pipeline, including the models used for systematization, test generation, rollout, and judging, as well as the rubric criteria and test design dimensions. This specification binds the reported rate to the specific choices used to produce it, enabling conditional reproducibility and the tracing of differences across audits.

3. Case Study: Multiverse Analysis of Conversational Deception

To demonstrate the pipeline, the authors conducted a multiverse analysis on conversational deception, fixing the evaluated system to GPT-5.5 while varying the measurement specifications.

Experimental Setup

• Baseline: GPT-5.5 system, Grok-4.3 simulated user, GPT-5.5 judge, 10-criterion rubric, four interaction conditions (material reliance, false premise, consistency challenge, accountability), and 120 total test cases.
• Variables: The study varied the simulated user (Grok-4.3 vs. GPT-5.5), the judge (Grok-4.3, GPT-5.5, Opus-4.7), the interaction condition, and the evidentiary standard (strict vs. loose).

Key Results

1. Sensitivity to Measurement Choices: The reported compliance rate varied substantially based on measurement choices, even with the same underlying system.
   • Simulated User: Replacing the Grok-4.3 user with GPT-5.5 increased the reported compliance rate from 82% to over 90%.
   • Judge: Substituting the judge changed the rate from 80% (Opus-4.7) to 95% (Grok-4.3). The GPT-5.5 baseline judge reported 82%.
   • Evidentiary Standard: Changing the standard from "loose" (flagging plausible evidence) to "strict" (requiring unambiguous evidence) shifted the rate from ~80% to >90%.
2. System Rankings: The relative ranking of different GenAI systems (GPT-5.5, Opus-4.7, Grok-4.3) was not stable across different judges. No single ranking held across all three judges; for instance, Opus-4.7 was ranked highest by the Grok-4.3 judge but lowest by the GPT-5.5 judge in certain configurations.
3. Noise Floor: Conditional reproducibility checks (rerunning the same specification) showed a noise floor of roughly 82%–87%, indicating that shifts larger than this range are likely due to substantive specification changes rather than stochastic variation.

4. Key Contributions

1. A Specification-Driven Pipeline: ASSERT provides a mechanism to record measurement choices in a written specification and execute the corresponding audit, supporting reproducibility and interpretability.
2. Empirical Evidence of Specification Sensitivity: The case study demonstrates that reasonable alternative choices in dialogue setup, simulated users, judges, and evidentiary standards can significantly alter reported rates and reorder system rankings.
3. Multiverse Analysis Framework: The paper applies a multiverse analysis approach to GenAI auditing, testing whether claims about system behavior hold up across defensible measurement choices rather than relying on a single static benchmark.

5. Significance and Claims

The paper argues that a single point estimate from a GenAI audit can be misleading if the underlying specification is opaque. The authors claim that:

• Comparisons require robustness: Claims about system performance differences should only be trusted if they remain robust across reasonable alternative specifications. If changing an instrument (e.g., the judge) reverses which system appears better, the ranking cannot be attributed to the systems alone.
• Transparency is necessary but insufficient: While transparency is required for meaningful safety measurement, it is not enough; the measurement choices must be explicit, inspectable, and open to scrutiny.
• The role of the specification: ASSERT helps researchers systematize vague background concepts into inspectable, operationalized activities. The reported rate answers to the edited specification, not just the raw data.
• Decision-making: When gaps in performance inform deployment decisions, a gap smaller than the shift induced by reasonable alternative specifications should not be treated as decisive.

The authors position ASSERT not as a replacement for existing benchmarks but as a complementary approach that makes the "multiverse" of measurement choices explicit, allowing stakeholders to understand the stability and limits of reported safety metrics.