Technical Summary: A Benchmark for Strategic Auditee Gaming Under Continuous Compliance Monitoring

Problem Statement

Emerging regulations, specifically the EU AI Act and the Digital Services Act (DSA), mandate continuous post-deployment compliance monitoring for AI systems and large online platforms. Unlike prior work on strategic classification and audit games, which typically model one-shot interactions, these new regimes create a longitudinal audit relationship. In this setting, a regulated firm (the auditee) reports outcome metrics over multiple rounds, while an auditor (regulator) selects which rounds to scrutinize based on a temporal policy.

The paper identifies a critical gap: existing literature largely assumes "honest-but-curious" auditees or single-round interactions. It fails to model how an adaptive auditee, possessing discretion over reporting timing, metric definitions, and population coverage, might strategically "game" a continuous audit system. Empirical evidence from the DSA Transparency Database already shows inconsistencies in moderation reporting and "audit blind-spots" created by API restrictions, suggesting that strategic adaptation is a real-world phenomenon requiring formal analysis.

Methodology

The authors formalize continuous compliance auditing as a $T$-round Stackelberg game:

• The Auditor (Leader): Commits to a temporal policy $\pi$ at $t=0$ (e.g., a fixed schedule or a randomized distribution) that determines which rounds are audited. The decision rule on audited rounds is a noise-aware Wald-style threshold, where the acceptance band widens for smaller sample sizes ($n_t$) to account for statistical uncertainty.
• The Auditee (Follower): Observes the auditor's committed policy and chooses a reported metric $\hat{m}_t$ and a sample size $n_t$ for each round. The auditee can adapt strategies based on the audit schedule and detection history.

The Benchmark Artifact

To support empirical study, the authors contribute a reproducible Python simulator featuring:

1. Five Auditee Strategies:
   • Delay: Lagging unfavorable metric shifts.
   • Drift: Adding a fixed bias to the true metric.
   • Cherry-pick: Selecting the maximum from $K$ stochastic metric estimates.
   • Attrition: Reducing sample size ($n_t$) when metrics are unfavorable to widen the auditor's noise tolerance, while applying a small drift.
   • OffAuditDrift: An audit-aware strategy that reports honestly on audited rounds and drifts on all others, exploiting the auditor's committed schedule.
2. Five Auditor Policies:
   • Static Baselines: One-shot, Periodic, and Scheduled-random.
   • Extensions: Min-sample floor (enforcing a minimum $n_t$) and Suspicion-escalation (increasing audit frequency based on observed gaps).
3. Evaluation Metrics:
   • Gaming Gap ($G$): Mean difference between reported and true metrics on audited rounds.
   • Time-to-Detection ($\tau_d$): Rounds until a gap is flagged.
   • Harm Decomposition: A non-additive split into Welfare Loss ($W$) (harm to individuals inside the verified report on unaudited rounds) and Coverage Loss ($C$) (harm to individuals excluded from the report due to sample attrition).

Key Contributions

1. Formalization of Continuous Auditing (C1)

The paper establishes the longitudinal complement to one-shot strategic classification, modeling the interaction as a $T$-round Stackelberg game where the auditee has discretion over timing, definitions, and coverage.

2. Non-Additive Harm Decomposition (C2)

The authors introduce a framework separating harm into $W$ (audited surface) and $C$ (excluded surface). They argue these are not summands of a total loss; rather, the strategic objective is often the reallocation of harm from the regulator-accountable surface ($W$) to the regulator-invisible surface ($C$). Attrition does not necessarily reduce total harm; it shifts it to a "regulator-invisible" domain.

3. The Cover Regime Observation (C3)

Observation 4.1 identifies a structural trade-off in any noise-aware static-auditor design (the "cover regime"). For a specific interval of drift magnitudes ($\Delta$), a static temporal-coverage policy faces a binding trade-off:

• Coverage Failure: If the audit set is not exhaustive ($\mathcal{A}_\pi \subsetneq \{0, \dots, T-1\}$), an auditee can drift on unaudited rounds undetected.
• Granularity Failure: If the audit set is exhaustive, an auditee can reduce sample size ($n_t$) to widen the noise envelope, allowing a drift within the "cover regime" to escape detection.
  The observation proves that a static policy cannot close both gaps simultaneously. It identifies a sample-size-aware static rule (Min-sample floor) and a history-conditioned policy (Suspicion-escalation) as orthogonal exits from this regime.

4. Empirical Validation (C4)

Using parameters calibrated to DSA Transparency Database statistics, the simulator demonstrates:

• Attrition Effect: The Attrition strategy reduces $W$ by ~90% while creating significant $C$, effectively hiding harm from the regulator.
• Orthogonal Exits: The Min-sample floor policy successfully detects Attrition (closing the granularity failure) but fails against OffAuditDrift (coverage failure). Conversely, Suspicion-escalation detects naive Drift faster but fails against Attrition.
• Audit-Aware Exploitation: The OffAuditDrift strategy defeats all static policies by reporting honestly only on known audit rounds, validating the "Coverage Failure" case of the Cover Regime.

5. Calibration and Reproducibility (C5)

The benchmark is calibrated to summary statistics from published DSA audits (e.g., drift envelopes, sample size ratios) and is provided as a pip-installable package with deterministic seeds and a small, extensible Python interface.

Results and Significance

The paper claims that continuous auditing creates a distinct class of strategic gaming where the auditee can exploit the longitudinal nature of the relationship to shift harm from visible to invisible surfaces.

• Structural Limitation: The "Cover Regime" demonstrates that static temporal policies are inherently vulnerable. Increasing audit frequency does not solve granularity failures (sample size manipulation), and enforcing sample sizes does not solve coverage failures (strategic timing).
• Strategic Adaptation: The OffAuditDrift strategy proves that if the auditor commits to a deterministic or known-random schedule, a rational auditee can guarantee zero detection on audited rounds while maximizing harm on unaudited rounds.
• Policy Implications: The results suggest that effective continuous auditing requires moving beyond static temporal schedules. The paper proposes that sample-size constraints and history-conditioned escalation are necessary minimal extensions, though neither is sufficient on its own against a fully adaptive adversary.

The significance of the work lies in providing the first runnable, game-theoretic environment for testing hypotheses about adaptive auditee behavior in regulatory contexts, moving beyond the "honest-but-curious" assumption prevalent in current AI fairness literature. It highlights that without adaptive auditor mechanisms, continuous compliance may create a false sense of security while harm is systematically reallocated to unmonitored populations.