Technical Summary: Self-CTRL: Self-Consistency Training with Reinforcement Learning

Problem Statement

Large Language Models (LMs) frequently produce explanations that are unfaithful to their actual decision-making processes. Models often state principles (e.g., "I will not generate discriminatory content") that they fail to follow when responding to specific user requests. This lack of self-consistency undermines user trust, complicates auditing, and hinders the ability of developers to monitor model alignment. Standard training optimizes models to produce plausible responses to individual prompts in isolation, without enforcing a relationship between meta-level explanations (what the model says it will do) and object-level behaviors (what the model actually does). Consequently, models can learn to answer meta-questions plausibly without those answers predicting their behavior on related inputs.

Methodology: Self-CTRL

The paper introduces Self-Consistency Training with Reinforcement Learning (Self-CTRL), a framework that explicitly optimizes LMs for consistency between their self-explanations and their behaviors. The method treats explanation and behavior as paired outputs generated from distinct but related inputs:

• Meta-level input ($x_{meta}$): A query asking the model to describe its behavior on a class of inputs (e.g., "Describe how you handle requests involving discrimination").
• Object-level input ($x$): A concrete instance from that class (e.g., a specific request to generate discriminatory text).

The model separately samples an explanation ($y_{meta}$) and a behavior ($y$). An external consistency function, $\phi(y_{meta}, y; x_{meta}, x)$, scores whether the explanation correctly predicts or justifies the behavior.

Optimization Framework

Self-CTRL formulates consistency as a reinforcement learning problem. The objective is to minimize a loss function derived from the negative expected consistency score:
$$L(x, x_{meta}) = -\mathbb{E}_{y, y_{meta}}[\phi(y_{meta}, y; x_{meta}, x)]$$

The framework supports two complementary learning directions, which can be optimized independently or jointly via a convex combination ($\lambda$):

1. Explanation Training ($\lambda = 0$): Updates the model to generate explanations that better predict its existing behavior. This produces more faithful self-reports without necessarily changing the model's actions.
2. Behavior Training ($\lambda = 1$): Updates the model's behavior to better satisfy its stated explanations. This aligns the model's actions with its principles.
3. Mixed Training ($0 < \lambda < 1$): Simultaneously optimizes both explanations and behaviors.

The training utilizes a GRPO-style estimator, sampling groups of candidate explanations and behaviors. A reference behavior (or explanation) is fixed, and rewards are assigned based on the consistency score $\phi$.

Instantiation of $\phi$

The consistency function $\phi$ is instantiated differently depending on the domain:

• Formal/Probabilistic Domain: $\phi$ is computed as the log-likelihood of the observed behavior under the probabilistic distribution described by the explanation (e.g., a stated coin bias predicting a sequence of coin flips).
• Constitutional/Natural Language Domain: Since natural language rules do not induce direct likelihoods, $\phi$ is instantiated using an LM Jury. Multiple LLMs acting as judges with different ethical frameworks (e.g., utilitarianism, deontology) evaluate whether the behavior complies with the stated rule. The final score is the mean of these juror verdicts.

Key Contributions and Experiments

The paper validates Self-CTRL in two distinct domains:

1. Probabilistic Reasoning with Formal Explanations

• Setup: Models are trained to generate samples from biased coins (behavior) and report the bias as a program (explanation).
• Findings: Initially, models could generate calibrated samples but failed to articulate the underlying bias (low $R^2$). After Self-CTRL explanation training, the correlation between articulated bias and empirical behavior improved significantly ($R^2$ from 0.24 to 0.64 on held-out distributions). This matched the generalization performance of a model trained with direct ground-truth supervision, demonstrating that consistency training can recover latent structure without explicit labels.

2. Constitutional AI with Natural Language Explanations

• Setup: Models are trained to describe principles for handling safety-relevant requests (e.g., hate speech, discrimination) and then respond to those requests.
• Findings:
  • Explanation Training: Improved the ability of a third-party auditor to predict model refusals based on the model's stated rules. Refusal prediction accuracy by an auditor model increased from 36% to 92%.
  • Behavior Training: Improved alignment by reducing the HarmBench attack success rate from 15.0% to 0.5% without substantially increasing over-refusal on harmless prompts.
  • Mixed Training: Achieved a Pareto-optimal trade-off, simultaneously improving simulatability (explanations predicting behavior) and safety (reducing harmful outputs).

Results and Significance

The paper claims that Self-CTRL provides a general recipe for training AI models to be safer, more transparent, and more controllable. Key results include:

• Faithfulness: The method successfully aligns what models say with what they do, addressing the "unfaithful explanation" problem.
• Scalability: Because consistency is evaluated using the model itself (or an LM jury) rather than human labeling, the method scales to generate large amounts of paired evidence.
• Native Interface: Self-explanations are elicited through the same channel users interact with, making them directly useful for trust and debugging.
• Generalization: The method demonstrates out-of-context generalization, recovering latent biases and refining refusal boundaries on held-out data without direct supervision.

The authors position Self-CTRL as a complement to external interpretability. Unlike external audits, it trains models to make their own behavioral regularities explicit in natural language. The paper modestly notes limitations, such as the potential for models to converge to trivial fixed points (e.g., vague "be safe" rules) if the explanation space is not constrained, and the reliance on the quality of the consistency judge (the LM jury). However, the experiments suggest that with careful design (structured prompts, diverse juries), these issues can be mitigated.