Technical Summary: Improving Labeling Consistency with Detailed Constitutional Definitions and AI-Driven Evaluation

Problem Statement

Automated labeling pipelines, particularly in content moderation, rely on "golden labels" derived from written category specifications. However, standard definitions (often one or two sentences) lack the granularity required to resolve edge cases consistently. When definitions are too brief, human annotators and Large Language Models (LLMs) fall back on their internal training priors or intuition rather than the written rules. This leads to significant drift in labeling accuracy and consistency, especially when different models or annotators interpret the same short definition differently. Furthermore, human working memory limits prevent annotators from retaining complex, rule-heavy specifications, causing them to compress specifications into heuristics.

Methodology

The authors propose an AI-driven workflow centered on Constitutional Specifications. Unlike prior "Constitutional AI" work focused on behavioral principles or runtime enforcement, this approach treats the constitution as a detailed, per-category operational specification designed to produce consistent golden labels for downstream processes.

1. Constitution Structure

Each category (e.g., Harassment, Hate Speech, Non-Violent Crime) is defined by a structured Markdown document (~300+ lines) containing:

• Definition & Decision Logic: Explicit rules for triggering a positive classification.
• Required Elements: Specific criteria that must all be met.
• Routing & Boundary Notes: Rulings on adjacent categories and edge cases (e.g., distinguishing political criticism from harassment).
• Worked Examples: Step-by-step reasoning for positive and negative cases.
• Dual-Axis Formulation: Two independent binary labels per category:
  • Intent: An attempt to cause or obtain harm.
  • Content: Harmful material appearing in the conversation.
  • Note: Evaluation occurs over the full conversation rather than individual messages to capture multi-turn attacks and gradual escalation.

2. Authoring and Refinement Pipeline

The workflow follows a "human-directs-AI-executes" pattern:

• Authoring: Humans identify gaps (e.g., a misclassification or a new attack pattern) and direct AI to revise the constitution.
• Cross-Model Validation: The constitution is applied to production conversations using multiple frontier LLMs from different vendors (e.g., OpenAI, Anthropic, Google). Disagreements between these models are treated as diagnostic signals indicating ambiguous or incomplete sections of the specification.
• Refinement Loop: An agentic skill traces disagreements to specific constitutional sections, drafts targeted patches, and submits them for human review. Once merged, the constitution is re-validated. This iterative process converts implicit rulings into explicit ones.

3. Evaluation Setup

The approach was evaluated on three categories: Harassment, Non-Violent Crime, and Hate Speech.

• Datasets: HarmBench (for measuring definition divergence and rater agreement) and WildChat (for evaluating cross-model stability under realistic production conditions).
• Baselines: Six definition sources were compared, including four published taxonomies (OpenAI, Llama Guard 3, MLCommons AILuminate, AEGIS), a paragraph-level definition, and the final refined constitution.
• Metrics: Pairwise disagreement rates per 1,000 conversations and three-rater unanimity (comparing three human annotators vs. three frontier LLMs reading the same constitution).

Key Results

1. Reduction in Cross-Model Disagreement

The constitutional approach significantly reduced inconsistency between different LLMs.

• On WildChat, cross-model disagreement was reduced by up to 57× compared to paragraph-level definitions.
• For frontier models (e.g., GPT-5.4, Opus 4.6, Gemini 3.1), disagreement rates dropped to under 3 per 1,000 conversations, whereas paragraph definitions resulted in rates ranging from 2 to 66 per 1,000.
• The reduction was most dramatic for broad categories like Non-Violent Crime, where short definitions failed to resolve subtypes, whereas the constitution provided explicit boundary rulings.

2. Superiority of LLMs over Humans in Applying Specifications

Contrary to the assumption that humans are the gold standard for applying rules, the study found that three frontier LLMs reading the same constitution achieved higher unanimity than three human annotators reading the same document.

• Non-Violent Crime: LLM non-unanimity was 84.2 per 1,000 vs. 301.0 for humans.
• Harassment: LLM non-unanimity was 37.9 vs. 43.4 for humans.
• Analysis: Human annotators frequently failed by treating multi-label classification as single-label (choosing one category over another) or by flagging surface-level harm without applying the specific decision logic. LLMs, conversely, evaluated each category independently and consistently applied the full document context.

3. Diagnostic Value of Disagreement

Cross-model disagreement served as an effective diagnostic tool. The majority of residual disagreements among frontier models were not instruction-following failures but genuine ambiguities where the constitution applied multiple provisions without a shared prior on user suspicion. These cases were used to refine the "conservatism stance" and add specific boundary rulings.

Significance and Claims

The paper claims that structured per-category constitutions paired with LLM consensus panels offer a scalable solution for producing accurate, consistent, and auditable golden labels.

• Shift in Human Role: The human role shifts from per-conversation annotation (which is prone to drift and cognitive overload) to specification authoring and disagreement triage.
• Operational Precision: The dual-axis (intent/content) formulation provides downstream consumers (e.g., safety teams, customer support) with distinct signals. For instance, a system can log "intent without content" to investigate repeat abusers even if the model refused the request, a distinction lost in single-label systems.
• Scalability: The approach demonstrates that AI can maintain and refine complex specifications that exceed human working memory capacity, ensuring that the "golden label" remains stable across different models and over time.

The authors note limitations, including the need for better filtering of AI-generated patches before they enter a CI/CD pipeline and the observation that current models may struggle with extremely long or complex exception lists. However, the core finding is that tighter specification, enforced by AI-driven validation, yields more consistent safety outcomes than relying on short definitions or human intuition.