Technical Summary: How LLMs Audit Each Other

1. Problem Statement

The paper addresses a foundational gap in the emerging field of automated Large Language Model (LLM) evaluation: the assumption that LLM auditors produce stable, consistent outputs regardless of contextual variables. Specifically, the study investigates two critical failure modes that have received little empirical scrutiny:

1. Auditor Identity-Disclosure Bias: Whether an auditor's analysis changes when it knows the identity of the model it is evaluating (e.g., brand recognition) versus when it is blind to that identity.
2. Cross-Lingual Auditor Instability: Whether an auditor produces diagnostically compatible outputs when the language of the auditing prompt mismatches the language of the evidence being assessed.

The authors challenge the prevailing assumption that LLMs function as stable "judges" in preference learning, adversarial deliberation, or ethical consistency testing, noting that prior work has largely focused on output quality (e.g., position bias, verbosity) rather than the consistency of the analytical behavior across these specific conditions.

2. Methodology

The study employs a rigorous experimental design involving nine commercial models (ChatGPT, Claude, Copilot, DeepSeek, Gemini, Grok, Meta AI, Perplexity, and Qwen) across three distinct components.

Core Design Principle: Independent Clean Sessions

A defining feature of the methodology is the use of "clean, independent chat sessions" with no cross-contamination between conditions.

• Evidence Corpus: The target evidence (CECT - Contextual Ethical Consistency Test) was generated independently in English and Spanish by the target models. These were not translations of one another but independent productions, ensuring that cross-lingual divergences in the evidence were production artifacts, not translation errors.
• Session Isolation: Every audit (blind vs. revealed, English prompt vs. Spanish prompt) occurred in a separate session. Auditors had no access to previous audit outputs or parallel evaluations.

Experimental Components

1. Component 1 (Single-Auditor Peer Review, N=9 pairs): Pairs of models evaluated each other's CECT transcripts under two conditions: Blind (identity unknown) and Revealed (identity disclosed).
2. Component 2 (Cross-Lingual Self-Audit, N=17 usable sessions): Models audited their own prior CECT outputs in two variants: Variant A (English prompt/Spanish evidence) and Variant B (Spanish prompt/English evidence).
3. Component 3 (Dual-Auditor Review, N=6 targets): Two independent architectures (e.g., Claude and DeepSeek) evaluated the same target simultaneously in clean sessions to test for stabilization effects.

Coding and Analysis

The study explicitly excluded numeric self-assigned scores from its primary conclusions, deeming them unreliable. Instead, analysis focused on the justificatory text using a pre-registered codebook (F1–F6) to measure:

• Option tracking specificity
• Contradiction counts
• Evidence citation retention
• Severity per subsection
• Self-admission accuracy

3. Key Contributions

The paper operationalizes "auditor bias" not as a single construct but as five distinct, mechanistic phenomena:

1. Register Collapse: A shift from operational analysis (tracking decision options, pressure types) to descriptive commentary (personality traits, tone) when the target identity is revealed.
2. Selective Evidence Loss: Specific, well-evidenced findings present in the blind condition vanish in the revealed condition without refutation or acknowledgment.
3. Reconstructive Instability: Basic factual claims about the audited transcript (e.g., baseline options) differ factually between blind and revealed conditions without new evidence.
4. Scope Narrowing: A reduction in the total number of distinct findings reported in the revealed condition, with omitted findings unaddressed.
5. Asymmetric Severity Redistribution: The intensity of critical analysis shifts between subsections (e.g., language sections) while the aggregate severity score remains stable, masking the redistribution.

Additionally, the paper contributes:

• Empirical Evidence of Self-Scoring Failure: Demonstrating that numeric self-scores fail to detect four of the five documented mechanisms.
• Cross-Lingual Instability Characterization: Showing that instability is model-specific (varying by architecture) rather than language-specific.
• Triangulation Framework: A method for validating findings through independent parallel evaluation.

4. Results

Mechanism Frequency (Component 1)

In the 9 peer-review pairs, five distinct bias mechanisms were observed (one instance each), while 5 pairs remained stable or sharpened under identity disclosure.

• Key Finding: Identity disclosure does not universally cause leniency; in some cases (e.g., Grok, Qwen), it acted as disambiguating information. However, when bias occurred, it manifested through the five specific mechanisms above.
• Numeric Score Failure: In cases of Selective Evidence Loss, Reconstructive Instability, Scope Narrowing, and Asymmetric Redistribution, the auditors' self-reported numeric scores remained identical (e.g., 3/3) despite qualitative outputs changing drastically.

Cross-Lingual Instability (Component 2)

Three models (Grok, Perplexity, Qwen) exhibited "Max Divergence" or significant divergence between language variants.

• Directionality: The direction of instability was not uniform. Grok and Perplexity were more rigorous under English prompts, while Qwen was more rigorous under Spanish prompts. This rules out a simple "dominant language" hypothesis, indicating the effect is model-specific.

Dual-Auditor Stabilization (Component 3)

In 5 of 6 targets, independent parallel evaluation by two different architectures produced stable or improved agreement post-reveal.

• Anomaly: In the Meta AI target, auditors diverged directionally, a phenomenon a single-auditor design would have missed.
• Conclusion: Dual-auditor designs offer a structural mitigation for identity-disclosure bias.

5. Significance and Claims

The paper asserts that the use of LLMs to evaluate other LLMs is methodologically legitimate but requires structural safeguards currently absent in practice.

• The Self-Scoring Blind Spot: The primary methodological claim is that self-reported numeric scores are unreliable indicators of auditor stability. Because mechanisms like evidence loss and scope narrowing preserve the structure of the report while altering the content, aggregate scores fail to signal that the audit has become less rigorous or factually inconsistent.
• Safety Implications: The behaviors described (normative drift, justificatory decoupling) fall outside the scope of existing vulnerability programs and evaluation frameworks. Relying solely on automated LLM-to-LLM evaluation without independent human verification or multi-architecture triangulation creates a documented structural blind spot in AI safety and regulatory compliance.
• Actionable Mitigation: The study proposes that organizations deploying LLM auditors must implement:
  1. Independent parallel evaluation by at least two architectures from different developer provenance.
  2. Human spot-checks using explicit content-level criteria (not just scores).
  3. Per-model empirical testing of cross-lingual stability before multilingual deployment.

The authors conclude that an LLM auditor reporting stability via a numeric score is not evidence that the audit was stable; only a qualitative comparison of the justificatory text across conditions can establish true stability.