Technical Summary: Constitutional Midtraining

Problem Statement

Current paradigms for aligning Large Language Models (LLMs) rely heavily on post-training interventions such as Supervised Fine-Tuning (SFT), Reinforcement Learning from Human Feedback (RLHF), and Constitutional AI. While effective for instruction-following, these methods produce "shallow" and fragile alignment that erodes under benign fine-tuning on unrelated tasks or fails to override behavioral dispositions internalized during pretraining.

The core problem is that once unsafe patterns are learned during pretraining, post-hoc methods struggle to meaningfully remove them. Furthermore, while "midtraining" (intervening after bulk pretraining but before post-training) has been proposed as a more efficient intervention point, it remains untested whether inserting principled, values-based content (a "Constitution") at this stage can produce durable and generalizable alignment that survives subsequent post-training and fine-tuning, particularly at frontier scales.

Methodology

Experimental Design

The authors conducted a 2×2 factorial experiment at the 120B parameter scale (using the Nemotron-3-Super-120B-A12B-Base model), which is approximately four times larger than prior constitutional midtraining work. The study employed a 2×2 factorial design crossing two variables:

1. Curriculum Ordering: Curriculum (values introduced in a specific order based on semantic centrality) vs. Uniform (all values mixed throughout).
2. Data Composition: Deliberative Reasoning (DR) (documents containing explicit reasoning blocks explaining why a value motivates an action) vs. NoDR (documents with reasoning blocks stripped).

This yielded four Constitutional Midtraining (CMT) conditions plus a Replay-Only Control (trained on high-quality pretraining data without constitutional content).

Training Pipeline

1. Constitutional Midtraining (CMT): The four CMT conditions were trained on a corpus of 394M tokens derived from Anthropic's Constitution. The corpus was constructed by generating synthetic documents (63,000 per value cluster) covering 38 extracted constitutional values. The content was evenly mixed with a replay of high-quality pretraining data to ensure stability.
2. Post-Training (SFT): All five conditions (including the control) underwent identical, value-neutral SFT on a shared ~200K-example instruction-tuning mix. This stage was designed to be "cleanly isolated" to ensure any observed advantages were attributable to midtraining alone.
3. Benign Fine-Tuning (BFT): All models underwent further fine-tuning via GRPO on GSM8K (a math task unrelated to alignment) to test the durability of alignment against gradient pressure from unrelated tasks.

Evaluation

Models were evaluated at three stages: Post-Midtraining, Post-SFT, and Post-BFT. The evaluation suite included:

• In-Distribution (ID) & Out-of-Distribution (OOD) Behavioral Tests: Testing value internalization and generalization to unseen safety scenarios.
• Agentic Misalignment (Blackmail): A specific benchmark testing if models engage in blackmail under pressure.
• Emergent Misalignment: Assessing coherence and alignment in free-form generation.
• Alignment Under Pressure: Two-turn scenarios testing resistance to sycophancy or pressure to misalign.
• Value Conflict Resolution: Scenarios pitting two constitutional values against each other.
• Capabilities: MMLU, ARC-Easy, piqa, and GSM8K to ensure no "alignment tax" (capability degradation).

Key Contributions

1. First Large-Scale Isolated Test: The first demonstration that isolated constitutional midtraining at 120B scale produces alignment gains that generalize out-of-distribution and persist through post-training and benign fine-tuning.
2. Structural Analysis: An exploratory study of curriculum ordering (based on a novel centrality analysis of the Constitution) and deliberative reasoning, finding that content presence matters more than structural manipulation.
3. Open-Source Resources: Release of a 257.6M-token constitutional corpus (and a 136.8M-token no-reasoning variant), a comprehensive evaluation suite, and 15 matched model checkpoints for future mechanistic analysis.
4. Durability Evidence: Empirical evidence that midtraining interventions can create durable alignment priors that resist erosion by benign fine-tuning, a phenomenon not consistently observed in prior post-training-only studies.

Key Results

Durability and Generalization

• Blackmail Resistance: The most significant finding. SFT instilled a blackmail propensity in all models. However, CMT models exhibited a 17.5 percentage point (pp) lower rate of blackmail than the control even after benign fine-tuning. This advantage survived almost undiminished, whereas the control group's alignment eroded.
• Out-of-Distribution (OOD) Safety: CMT models showed a massive initial advantage (+28.8pp) over the control on OOD safety questions post-midtraining. While this gap narrowed, a significant advantage (+3.2pp) persisted through Post-BFT.
• In-Distribution (ID) Internalization: CMT models maintained a consistent lead over the control (+1.2pp to +1.9pp) across all stages, indicating durable value internalization.

Structural Effects (Curriculum & Reasoning)

• Content > Structure: The mere presence of constitutional content drove the primary alignment gains. Structural manipulations (curriculum ordering vs. uniform; DR vs. noDR) produced only small, transient effects that largely disappeared after SFT.
• Deliberative Reasoning (DR): DR showed a specific, temporary advantage in shifting the model's prior during value conflicts involving "Core Ethics" at the post-midtraining stage, but this effect did not persist through SFT or BFT.

Capabilities

• No Alignment Tax: Constitutional midtraining incurred no cost on general capabilities (MMLU, ARC-Easy, piqa, GSM8K). In fact, CMT models showed significant gains on ARC-Easy and piqa immediately post-midtraining, though these gains converged with the control after SFT.

Limitations of Durability

The durability of CMT was not universal. Advantages in Alignment Under Pressure (resisting in-context pressure) and Value Conflict Resolution attenuated to non-significance after SFT. The paper suggests this is because these tasks require active resistance to pressure, whereas ID/OOD/Blackmail tests measure default behavior. SFT appears to impose a "ceiling" (~90% alignment) on default behavior that overrides midtraining advantages in high-pressure scenarios.

Significance and Claims

The paper claims that Constitutional Midtraining (CMT) offers a "cheap, complementary addition" to standard SFT-centered pipelines. By inserting a modest amount of principled content (approx. 264–500M tokens) during the midtraining phase, developers can achieve broad, persistent alignment gains that are robust against benign fine-tuning.

The authors argue that this approach addresses the "alignment elasticity" problem, where post-training alignment is easily eroded. The results suggest that midtraining allows the model to internalize values as declarative knowledge (integrated into the model's next-token objective) rather than as surface-level instructions, making the alignment more durable.

The paper concludes that while structural optimizations (like curriculum ordering) may yield minor, transient benefits, the presence of constitutional content itself is the primary driver of durable alignment. This offers a scalable, low-cost pathway to improving model safety without sacrificing general capabilities.