Three Models of RLHF Annotation: Extension, Evidence, and Authority

1. Problem Statement

Reinforcement Learning with Human Feedback (RLHF) and related methods (e.g., Direct Preference Optimization) are the dominant paradigms for aligning Large Language Models (LLMs) with human values. These pipelines rely on human annotators to rank or evaluate model outputs. However, the paper identifies a critical conceptual gap: the normative role of these annotators is rarely made explicit.

Pipeline designers make implicit assumptions about why annotators' judgments matter. Are annotators merely extending the designers' own preferences? Are they providing independent evidence about facts or community standards? Or do they possess legitimate authority to dictate system behavior as representatives of the public?

• The Consequence: When these roles are conflated or left undefined, it leads to incoherent pipeline designs, inconsistent validation metrics, and "failure modes" where the system fails to achieve its intended alignment goals. Furthermore, it obscures accountability, making it difficult to determine who is responsible for the model's outputs.

2. Methodology

The paper employs a conceptual and normative analysis rather than an empirical experiment. The methodology involves:

• Philosophical Frameworking: Drawing on political philosophy (authority, legitimacy) and epistemology (evidence, reason-giving) to categorize the normative role of annotators.
• Literature Survey: Analyzing landmark RLHF papers (e.g., Ouyang et al., Bai et al., Glaese et al.) and reform proposals to identify which conceptual model they implicitly rely on.
• Design Implication Mapping: Systematically deriving how each conceptual model dictates specific technical decisions in the RLHF pipeline, including:
  • Annotator Selection: Who should be hired? (Experts vs. random sample vs. designer proxies).
  • Instructions: How should prompts be worded? (e.g., "What fits policy?" vs. "What would the community think?").
  • Validation: How is quality measured? (Gold-standard labels vs. Inter-Annotator Agreement).
  • Aggregation: How are judgments combined? (Binary majority vs. scalar distributions).

3. Key Contributions: The Three Conceptual Models

The core contribution is the distinction of three mutually distinct models of annotation, each with specific technical implications:

A. The Extension Model

• Definition: Annotators act as proxies to extend the system designers' own preferences to novel cases they cannot adjudicate personally.
• Normative Basis: Mimicry of the designer's beliefs.
• Technical Implications:
  • Selection: Annotators are selected based on their ability to agree with designers (e.g., "soft rejection" if agreement < 75%).
  • Instructions: Framed around company policy or designer intent.
  • Validation: Relies on "Gold Standard" labels provided by designers; Inter-Annotator Agreement (IAA) is secondary.
  • Aggregation: Binary measures are sufficient; disagreement is treated as noise.

B. The Evidence Model

• Definition: Annotators provide independent evidence regarding facts (objective truths) or community beliefs (social facts).
• Normative Basis: Epistemic standing; designers must accept these judgments as reasons to believe a proposition is true.
• Technical Implications:
  • Selection: Based on expertise (for facts) or representativeness (for community preferences).
  • Instructions: Framed to elicit knowledge or perception of community standards (e.g., "Would the community find this offensive?").
  • Validation: Avoids Gold Standard labels; relies on IAA for facts, but rejects IAA for preference aggregation (as diversity is the signal, not convergence).
  • Aggregation: Scalar measures (distributions) are preferred for community preferences to capture heterogeneity; binary measures for objective facts.

C. The Authority Model

• Definition: Annotators possess legitimate practical authority (like a "mini-legislature") to determine system behavior, independent of whether their judgment is "correct" or "evidenced."
• Normative Basis: Democratic legitimacy and fairness; the "who" matters more than the "what."
• Technical Implications:
  • Selection: Must be a random, representative sample of the population.
  • Instructions: Framed as an exercise of judgment/authority (e.g., "Should the system be allowed to produce this?").
  • Validation: Rejects Gold Standard and strict IAA; focuses on procedural legitimacy.
  • Aggregation: Requires moral deliberation; Majority rule is a default but may require weighting for minority groups to ensure fairness.

4. Results and Analysis

The author surveys existing literature and identifies three major failure modes resulting from the conflation of these models:

1. Self-Defeat: Pipelines that mix incompatible models (e.g., selecting annotators for designer agreement while asking them to represent community standards) undermine their own goals.
2. Fragmentation: Vague instructions lead annotators to switch between models, creating a "hodgepodge" of data that confuses the reward model.
3. Misattribution (Responsibility Laundering): Designers claim to use the Authority or Evidence model to shift blame for harmful outputs to annotators, while secretly retaining control via the Extension model (e.g., filtering outliers).

Case Studies:

• InstructGPT (Ouyang et al.): Primarily Extension (annotators align with OpenAI researchers).
• Anthropic (Bai et al.): Primarily Evidence (annotators provide diverse views on "harmlessness" without filtering for agreement).
• Jury Learning (Gordon et al.): Authority (explicitly modeling group representation and legitimacy).

Constitutional AI: The paper analyzes Constitutional AI as a mixed model where the principles are set via Extension, but the application relies on Evidence or Authority, often leading to incoherence unless carefully managed.

5. Significance and Recommendations

The paper argues that no single pipeline fits all alignment tasks. The choice of model must be driven by the nature of the value being aligned:

• Style/Helpfulness: Best suited for the Extension model (property rights of designers).
• Factual Accuracy: Best suited for the Evidence (Facts) model.
• Offensiveness/Community Standards: Best suited for the Evidence (Preferences) model.
• Political Controversy/Democratic Legitimacy: Best suited for the Authority model.

Primary Recommendation:
RLHF pipeline designers should decompose annotation into separable dimensions and create heterogeneous pipelines. Instead of a single unified reward model, developers should:

1. Identify the specific value dimension (e.g., toxicity vs. tone).
2. Select the appropriate conceptual model for that dimension.
3. Tailor the selection, instructions, validation, and aggregation steps specifically to that model.

Broader Impact:
This framework transforms RLHF design from a purely technical engineering problem into a metaethical and political one. It provides a diagnostic tool to identify hidden inconsistencies in current systems and offers a path toward more legitimate, accountable, and effective AI alignment by clarifying the source of the "human" in Human Feedback.