Technical Summary: Decoy-Calibrated Failure Audits for Language Models

1. Problem Statement

The paper addresses the "reporting problem" in model failure audits. While audits can easily identify that a model fails, determining where failures concentrate and which explanations are credible enough to report is difficult. Auditors often test many candidate explanations (e.g., long inputs, indirect questions, specific metadata combinations). The risk is selection bias: the largest observed error lift may reflect a genuine failure mode, or it may simply be the most favorable result among many tried candidates (a "winner's curse").

Current slice-discovery methods (e.g., SliceFinder, SliceLine) and LLM-assisted debugging can generate many plausible hypotheses, but they lack a rigorous mechanism to distinguish between genuine failure modes and artifacts of the search process. The paper argues that an explanation found after many attempts requires stricter validation than one named in advance.

2. Methodology: Janus

The authors introduce Janus, a procedure designed not to generate new explanations, but to decide which proposed explanations are credible enough to report. The method relies on two main safeguards: decoy calibration and holdout replication.

Core Definitions

• Descriptor: An interpretable true/false property of an example (e.g., long_chain, indirect_query).
• Slice: The subset of examples where a descriptor is true.
• Lift: The difference in error rate between the slice and the complement ($err(g=1) - err(g=0)$).
• Decoy: A fake descriptor with the same prevalence (frequency of being true) as a real descriptor but with values randomly assigned to examples.
• Discovery Split: The data used to score descriptors against decoys.
• Holdout Split: A fresh, separate dataset used to verify survivors.

The Procedure

1. Candidate Library: The auditor starts with a fixed, frozen list of candidate descriptors ($G$) derived from domain knowledge, slice-mining, or LLM-assisted debugging.
2. Decoy Construction: For each real descriptor, Janus generates $K$ decoys by permuting the true/false values across the discovery examples while preserving the original prevalence. This creates an empirical "decoy floor" representing the maximum lift achievable by chance given the sample size and prevalence constraints.
3. Discovery Screening (Procedure A):
   • Descriptors are scored by their absolute lift on the discovery split.
   • Janus compares the number of real descriptors ($R$) and decoys ($D$) that exceed various lift thresholds.
   • It estimates the False Discovery Proportion (FDP) using the ratio: $\widehat{FDP}(\tau) = \frac{(L/K)D(\tau)}{\max(1, R(\tau))}$, where $L$ is the number of real descriptors and $K$ is the number of decoys.
   • The threshold is set to the most permissive value where the estimated FDP is below a target $q$ (e.g., 0.10).
   • Descriptors clearing this "decoy floor" become Discovery Survivors.
4. Holdout Confirmation:
   • Discovery survivors are re-evaluated on the holdout split.
   • A descriptor is confirmed only if:
     • It retains sufficient support (sample size).
     • Its absolute holdout lift exceeds a minimum threshold (e.g., $|lift| \ge 0.10$).
     • The sign of the lift matches the discovery lift.

This separation ensures that candidate descriptors may come from any source, but only those that beat the decoy floor and replicate on fresh data become audit findings.

3. Key Contributions

1. Formulation of the Reporting Problem: The paper explicitly defines the task of deciding which of many candidate descriptors to report, distinguishing between proposing explanations and validating them for reporting.
2. Janus Procedure: The introduction of a decoy-calibrated procedure that compares descriptor lift against fake descriptors of identical frequency and requires replication on held-out data.
3. Empirical Evaluation: The evaluation of Janus on one controlled audit with planted failure modes and two public benchmarks (MuSiQue and LongBench v2), demonstrating its ability to recover planted signals while rejecting plausible but unconfirmed findings.

4. Experimental Results

Controlled Audit (Planted Structure)

• Setup: 160 synthetic multi-table lookup tasks where a "long chain" failure mode (requiring multiple hops across tables) was intentionally planted.
• Result: Janus successfully recovered the planted failure. It confirmed six descriptors, including long_chain (lift +0.58) and its interactions with indirect queries and collisions.
• Observation: The model tended to stop partway through the lookup chain. Janus correctly identified this specific failure mode without flagging every high-error slice.

Natural Benchmarks (MuSiQue and LongBench v2)

• Setup: Audits on public benchmarks where plausible metadata descriptors exist (e.g., hop count, context size) but no ground-truth failure mode is guaranteed.
• Result: Janus reported zero confirmed descriptors for both benchmarks.
• Comparison: Baseline methods like SliceLine identified high-error subgroups (e.g., 85–93% error pockets in LongBench v2). However, these slices failed the decoy calibration or the holdout replication check.
• Ablation: Removing the decoy floor led to over-reporting (20 descriptors on LongBench v2). Removing the holdout gate allowed a single descriptor to pass despite its lift shrinking from 0.36 (discovery) to 0.05 (holdout), illustrating the "winner's curse."

5. Significance and Claims

The paper claims that a plausible error slice is not necessarily a reportable audit finding. The significance of Janus lies in its restraint:

• Separation of Discovery and Reporting: It separates the generation of hypotheses from the validation required for reporting.
• Empirical Calibration: It uses decoys to calibrate the "noise floor" of lift scores, acknowledging that large effects can arise from chance when searching large spaces.
• Replication as a Gate: It enforces that an explanation selected on one split must retain its direction and strength on another.

The authors emphasize that Janus is a reporting rule, not a complete explanation engine. It does not guarantee finite-sample False Discovery Rate (FDR) control in the formal statistical sense (like Benjamini-Hochberg) because it does not assume a fixed family of valid p-values or exchangeable knockoffs. Instead, it provides an empirical screen to filter candidates before a final holdout check. The paper concludes that candidates from any source (domain knowledge, slice-mining, LLMs) become audit findings only when they beat decoys and replicate on fresh data.