Technical Summary: Escaping the Agreement Trap

1. Problem Statement: The Agreement Trap

Automated content moderation systems are traditionally evaluated using agreement-based metrics (e.g., F1 score) against human labels. The authors argue this approach fails in rule-governed environments where multiple decisions can be logically consistent with the governing policy. In such "grey areas," human moderators resolve ambiguity using context, precedent, and interpretive norms, creating labels that reflect a specific resolution pathway rather than the underlying rule structure.

When a model learns from these labels, it reproduces the interpretive pathway. Consequently, agreement-based metrics penalize valid, policy-grounded decisions that diverge from the specific human label, conflating three distinct failure modes:

1. Model Error: The model violates the rules.
2. Moderator Divergence: The model and human chose different valid paths.
3. Policy Ambiguity: The rules themselves are underspecified.

The authors term this phenomenon the Agreement Trap. It leads to a systematic mischaracterization of rule adherence as error, particularly when the set of policy-defensible outcomes is set-valued (containing more than one element).

2. Methodology: The Defensibility Framework

To address this, the paper formalizes evaluation as policy-grounded correctness, shifting the focus from agreement with historical labels to logical derivability under an explicit rule hierarchy.

2.1 Formalization and Metrics

The framework evaluates a proposed decision $\hat{y}$ (e.g., REMOVE or APPROVE) against content $C$, a rule hierarchy $R$ (platform-wide $R_G$ and community-specific $R_C$), and a precedent corpus $P$. A decision is defensible if a valid logical derivation exists from $(C, R, P)$ to $\hat{y}$.

Decisions are classified into three levels:

• L1 (Robustly Defensible): An explicit rule directly and unambiguously authorizes the decision.
• L2 (Plausibly Defensible): Rules are genuinely ambiguous, but the decision is reasonably supported.
• L3 (Indefensible): No explicit rule authorizes the decision, or the reasoning invokes concepts absent from the rule set.

Two primary metrics are introduced:

• Defensibility Index (DI): The fraction of decisions that are L1 or L2. Target: $\ge 0.90$.
• Ambiguity Index (AI): The fraction of decisions where an "inverse check" (verifying if the opposite decision is also derivable) fires. Target: $\le 0.15$. AI is decomposed into platform-level ($A_{IG}$) and community-level ($A_{IC}$) underspecification.

2.2 Audit Implementation

The framework employs a structured audit using an Audit Model ($M_a$). Unlike standard "LLM-as-judge" approaches that evaluate subjective quality, $M_a$ acts as a constrained reasoning engine. It receives the content, rules, precedents, and the proposed decision, then generates a JSON reasoning trace:

1. Logic Chain: Inference steps.
2. Policy Citation ($\kappa$): Explicit rule reference (committed before the verdict).
3. Precedent Weight ($\omega$): Importance of past cases.
4. Inverse Check ($\iota$): Whether the opposite decision is also derivable.
5. Defensibility Level ($\xi$): L1, L2, or L3.

This ordering ensures the classification is grounded in an explicit governance reference, preventing the model from rationalizing a pre-determined output.

2.3 Probabilistic Defensibility Signal (PDS)

To estimate reasoning stability without additional audit passes, the authors introduce PDS, derived from the audit model's token log-probabilities. PDS extracts a three-component vector from a single forward pass:

1. $\lambda_\xi$ (Label Log-Confidence): Log-probability of the final defensibility level.
2. $H[\kappa]$ (Citation Span Entropy): Shannon entropy over the policy citation tokens. High entropy indicates uncertainty in rule selection.
3. $\sigma(\rho)$ (Inverse-Check Log-Odds): Log-odds of the inverse check firing. High values indicate the model is unsure if the decision is indefensible.

These components are combined into a scalar $S$ via calibration. The framework hypothesizes that variance in PDS is driven primarily by governance ambiguity rather than decoding noise.

3. Key Contributions

1. Formalization of Policy-Grounded Correctness: Introduction of DI and AI to separate policy-defensible decisions from indefensible ones, moving beyond agreement metrics.
2. Probabilistic Defensibility Signal (PDS): A zero-cost stability signal extracted from audit model log-probs that correlates with reasoning instability.
3. Governance Gate: An operational deployment mechanism that restricts automated enforcement to cohorts satisfying specific DI and AI thresholds, enabling principled automation coverage with risk reduction.

4. Empirical Results

The framework was validated on 193,000+ Reddit moderation decisions across multiple communities, audited against both platform-wide and community-specific rules.

4.1 The Agreement-Defensibility Gap

• Metric Divergence: There is a 33–46.6 percentage-point gap between agreement-based F1 scores and the Defensibility Index.
• False Negative Analysis: 79.8–80.6% of the model's "false negatives" (disagreements with human labels) were actually policy-grounded decisions (L1 or L2). This confirms the Agreement Trap: standard metrics penalize valid adherence to rules when they diverge from specific human interpretations.
• Root Cause: Only 19.4% of disagreements were true model errors (L3); the rest were implicit norm enforcement or policy ambiguity.

4.2 PDS and Stability

• Human Alignment: PDS correlates strongly with human defensibility ratings ($\rho = 0.66, p < 10^{-4}$).
• Ambiguity Detection: The inverse-check log-odds ($\sigma(\rho)$) is 63% higher in disagreement cases than agreement cases.
• Stability Analysis: Repeated sampling (1,000 replicates) shows that PDS variance is driven by governance ambiguity, not sampling noise. The ratio of instability between "flipping" and "stable" cases remains flat (~1.6x) across a tenfold temperature increase, supporting the governance ambiguity hypothesis.

4.3 Normative Underspecification Experiment

Auditing 37,286 decisions under three tiers of rule specificity (Title Only, Sidebar, Wiki) on r/AskReddit revealed:

• AI Reduction: Adding rule details reduced the Ambiguity Index by 10.8 percentage points (from 18.2% to 7.4%).
• DI Stability: The Defensibility Index remained stable (~97-98%), indicating the decisions were defensible regardless of rule detail; the uncertainty about their defensibility was what changed.
• Mechanism: Vague rules primarily create ambiguous (L2) decisions rather than indefensible (L3) ones.

4.4 Governance Gate Performance

A Governance Gate configured with DI $\ge$ 0.90 and AI $\le$ 0.15 achieved:

• 78.6% Automation Coverage across the fleet.
• 64.9% Risk Reduction (reducing the indefensible decision rate from 5.66% to 2.72% in the automated portion).
• The AI threshold was identified as the binding constraint, meaning selectivity is determined by policy ambiguity rather than reasoning validity.

4.5 Adversarial Robustness

In a "Spurious-Grounding Benchmark" (1,000 cases including adversarial attacks):

• Detection: The system detected 93.5% of action-flip attacks and 70.1% of hallucinated-grounding attacks.
• Escape Surface: 29.9% of attacks escaped, specifically those exploiting "policy penumbra" (subjective content rules). Crucially, the rules vulnerable to these attacks were the same ones identified by the Ambiguity Index as needing clarification, providing independent corroboration of the framework's diagnostic capability.

5. Significance and Claims

The paper claims that evaluation in rule-governed environments must shift from agreement with historical labels to reasoning-grounded validity under explicit rules.

• Reframing Ambiguity: The framework treats policy ambiguity not as a deficiency but as a deliberate design feature (Normative Underspecification) that preserves human judgment. The metrics quantify this elasticity to enable safe automation where defensibility is high.
• LLM as Reasoning Auditor: The work extends the "LLM-as-judge" paradigm from subjective quality assessment to rule-grounded logical verification. By treating rules as the external source of truth and the audit model as a constrained reasoning engine, the system operationalizes rule-consistency.
• Practical Deployment: The Governance Gate provides a principled mechanism for scaling automation while preserving auditability, demonstrating that LLM reasoning traces (specifically log-prob distributions) can serve as calibrated uncertainty signals for governance.

The authors conclude that the convergence of the Ambiguity Index, PDS variance, and adversarial escape surfaces points to a single underlying phenomenon: normative underspecification. The framework provides the infrastructure to identify and manage this underspecification rather than treating it as model error.