💬 NLP

CompliBench: Benchmarking LLM Judges for Compliance Violation Detection in Dialogue Systems

This paper introduces CompliBench, a novel benchmark and automated data generation pipeline designed to evaluate and improve LLM judges' ability to detect and localize compliance violations in enterprise dialogue systems, revealing that current models struggle with the task while demonstrating that small-scale models fine-tuned on the synthesized data achieve superior performance and generalization.

Original authors: Jingbo Yang, Guanyu Yao, Bairu Hou, Xinghan Yang, Nikolai Glushnev, Iwona Bialynicka-Birula, Duo Ding, Shiyu Chang

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Jingbo Yang, Guanyu Yao, Bairu Hou, Xinghan Yang, Nikolai Glushnev, Iwona Bialynicka-Birula, Duo Ding, Shiyu Chang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a very smart, well-read robot to work as a customer service agent for an airline or an insurance company. You give this robot a thick rulebook: "If a customer asks for a human, transfer them immediately," or "Always sound empathetic."

Now, imagine you need a supervisor to watch the robot's conversations and check if it followed the rules. You decide to use another super-smart AI (a "Judge AI") to do the supervising because it's faster and cheaper than hiring a human manager.

The Problem:
The authors of this paper asked: "Can we actually trust this Judge AI?"

They found that even the most advanced AI judges are terrible at this specific job. They often miss mistakes, or they get confused about which rule applies when. It's like hiring a referee who is so busy reading the rulebook that they miss the actual foul happening on the field.

The Solution: COMPLIBENCH
To prove this, the team built a new "test track" called COMPLIBENCH. Think of it as a driving simulator for AI agents, but instead of crashing cars, the agents "crash" by breaking company rules.

Here is how they built it, using some creative analogies:

1. The "Mad Libs" Rulebook Generator

Real customer service rulebooks are boring and specific. To test the AI, the researchers needed thousands of different scenarios.

What they did: They used AI to take real rulebooks and "remix" them. They created thousands of variations, like changing "Transfer the user" to "Ask the user three questions first."
The Analogy: Imagine a chef taking a standard recipe for chocolate cake and creating 1,000 variations: one with no sugar, one with salt instead of sugar, one where you have to stir counter-clockwise. They made sure these variations were distinct so the test wasn't repetitive.

2. The "Trap-Setter" (Adversarial Injection)

This is the coolest part. They didn't just want the AI to make obvious mistakes. They wanted to trick the Judge AI.

What they did: They created a "Villain AI" whose only job was to find the most subtle way to break a rule without the Judge AI noticing.
The Analogy: Imagine a security guard (the Judge) watching a museum. A thief (the Villain AI) tries to steal a painting.
- Easy theft: The thief runs out with the painting. The guard catches them immediately.
- Hard theft (The goal): The thief slowly swaps the painting with a fake one over 10 minutes, or hides it inside a coat. The guard should catch this, but often misses it.
- The researchers used this "Villain AI" to generate conversations where the agent breaks a rule in a sneaky way, creating a "Ground Truth" (the correct answer) that is hard to find.

3. The "Gold Standard" Labels

Because the researchers controlled the "Villain AI," they knew exactly where the mistake happened and which rule was broken.

The Analogy: In a normal classroom, a teacher has to grade a student's essay and guess if they followed the prompt. In this test, the teacher (the researchers) wrote the essay with the mistake included and marked the exact spot with a red pen. They know the answer key perfectly.

The Results: The "David vs. Goliath" Story

They ran the test with the world's biggest, most expensive AI models (the "Goliaths") and compared them to a tiny, custom-trained model (the "David").

The Goliaths (Big AI Judges): Even the smartest, most expensive AI models struggled. They often missed the sneaky violations or got confused about which rule applied. They were like a genius professor who is so smart they overthink simple things and miss the obvious.
The David (Small, Specialized AI): The researchers took a small AI model and trained it only on their "Villain AI" generated data. This tiny model beat the giants.
- Why? Because it learned the specific "language of mistakes" from the training data. It was like a specialized detective who has seen every type of trick in the book, whereas the giant AI was a generalist who knew everything about the world but nothing about these specific tricks.

Why This Matters

This paper is a wake-up call. Just because an AI is big and smart doesn't mean it's good at being a supervisor for other AIs.

For Businesses: If you use AI to check your customer service agents, you might be trusting a blindfolded referee. You need a specialized tool, not just a general smart AI.
For the Future: The best way to build a good supervisor AI isn't to make it bigger; it's to train it on high-quality, tricky examples of what not to do.

In a nutshell: The authors built a "trick question" exam for AI supervisors. They found that the biggest, most famous AIs failed the exam, but a small, specially trained AI aced it. They proved that to catch a cheater, you need a detective who knows exactly how cheaters think, not just a very smart person.

1. Problem Statement

As Large Language Models (LLMs) are increasingly deployed as task-oriented agents in enterprise environments (e.g., customer support, contact centers), ensuring their strict adherence to complex, domain-specific operational guidelines is critical. While "LLM-as-a-Judge" has emerged as a scalable solution for evaluating these agents, the reliability of LLM judges in detecting specific policy violations remains largely unexplored.

Current research faces two major bottlenecks in addressing this:

Lack of Fine-Grained Ground Truth: Unlike outlier detection in other fields, detecting conversation policy violations requires precise labels indicating which guideline was violated and at which specific turn. Obtaining this from real-world data is prohibitively expensive due to the need for domain expert annotation.
Difficulty in Synthetic Data Generation: Existing methods struggle to synthesize realistic agent interactions that contain specific, controllable violations without violating privacy regulations or producing unrealistic scenarios.

2. Methodology: The COMPLIBENCH Pipeline

The authors introduce COMPLIBENCH, a novel benchmark and a scalable, automated data generation pipeline designed to evaluate LLM judges on multi-turn dialogue compliance. The pipeline consists of three main stages:

A. Guideline Scaling (Data Expansion)

To address data scarcity, the authors construct a comprehensive guideline pool from real-world enterprise guidelines (Airline, Healthcare, Insurance).

Expansion: They use LLMs to expand seed guidelines into diverse Workflow Guidelines (step-by-step procedures for specific intents) and Condition Guidelines (trigger-action rules).
Quality Control: An iterative "judge-and-refine" loop ensures generated guidelines are non-overlapping and non-conflicting. A similarity-based filtering mechanism (combining embedding cosine similarity and LLM-based semantic scores) removes duplicate workflows, ensuring diversity.

B. Adversarial Violation Injection

To create realistic violations with ground-truth labels, the pipeline injects "flaws" into the agent's behavior:

Violation Variant Generation: For each oracle guideline, the system generates modified versions that are strictly incompatible with the original but remain realistic (e.g., omitting a required field or performing an incorrect action).
Adversarial Optimization: An automated Judge-and-Refine loop optimizes these variants. It employs two judges:
1. Content Consistency Judge: Ensures the modification results in a meaningful behavioral change (not just rephrasing).
2. Adversarial Compliance Judge: Attempts to detect the violation. Variants are retained only if the Content Judge confirms a change but the Compliance Judge fails to detect the violation. This ensures the benchmark contains challenging, "stealthy" violations.

C. Dialogue Synthesis and Labeling

Simulation: A multi-agent system simulates conversations. A User Simulator steers the dialogue to trigger specific conditions. A Selector Agent identifies the governing guideline for each turn, and an Assistant Agent generates responses based on the corrupted (violation-injected) guidelines.
Automatic Labeling: Because the system controls the injection, it automatically generates precise ground-truth labels ( $y_i = \{guideline, violation\_status\}$ ) for every turn.

3. Evaluation Metrics

The benchmark evaluates LLM judges on three specific metrics:

Strict Guideline Accuracy (SGA): Measures if the judge correctly identifies the governing guideline and correctly labels a compliant turn as non-violating.
Violation Detection Accuracy (VDA): Measures if the judge correctly flags a turn as a violation, regardless of whether it identified the specific guideline.
Conversation-Level Accuracy (CLA): Requires the judge to be correct on every turn within a conversation (both guideline identification and violation status).

4. Key Results

The authors evaluated state-of-the-art (SOTA) proprietary and open-source LLMs, as well as reward models, on the COMPLIBENCH dataset (318 multi-turn dialogues across Airline, Healthcare, and Insurance).

SOTA LLMs Struggle: Even the most advanced proprietary models (e.g., GPT-5, Gemini-3-pro, Claude-sonnet) perform poorly on Conversation-Level Accuracy (CLA), with scores ranging from ~11% to ~57%.
- Key Bottleneck: While models are decent at identifying guidelines on compliant turns (SGA), they fail significantly at Violation Detection (VDA). They tend to be too strict on compliant turns (hallucinating violations) and too loose on actual violations (missing them).
Reward Models Fail: Existing reward models (trained on single-turn preference data) perform even worse, with CLA scores below 38%, indicating they are ill-suited for long-context rule adherence.
Specialized Fine-Tuning Works: A small-scale model (Qwen3-8B) fine-tuned on the synthesized COMPLIBENCH data outperformed all general-purpose LLMs.
- It achieved the highest CLA (51.47% on Airline), surpassing Gemini-3-pro and GPT-5.
- It demonstrated strong generalization to unseen business domains, proving that task-specific supervision is more critical than model scale for this task.

5. Error Analysis

Manual analysis of GPT-5 errors revealed a systematic asymmetry:

On Compliant Turns: Judges are overly strict, often mis-attributing the guideline scope (Type 1 error: 57% of errors) or inventing dependencies that don't exist.
On Violated Turns: Judges are too lenient, accepting non-equivalent behaviors (e.g., handling a claim in chat when a transfer was required) or ignoring key evidence of missing actions.

6. Significance and Contributions

First Systematic Benchmark: COMPLIBENCH is the first benchmark to provide fine-grained, turn-level ground truth for compliance violation detection in multi-turn dialogues.
Scalable Data Synthesis: The paper proposes a novel, controllable pipeline for generating realistic, adversarial training data without human annotation, solving a major data scarcity issue in enterprise AI safety.
Training Robust Judges: The results demonstrate that small, specialized models fine-tuned on synthetic compliance data can outperform massive general-purpose models. This provides a viable path for developing robust "generative reward models" for enterprise compliance.
Open Source: The code and data are publicly available, fostering further research into reliable LLM evaluation for high-stakes domains.

In conclusion, the paper argues that while current LLMs are powerful, they lack the specific reasoning capabilities required for strict compliance monitoring. However, by using adversarial synthetic data to train specialized, smaller models, enterprises can achieve reliable, scalable compliance evaluation.