🤖 machine learning

Synthetic Persona Pretraining: Alignment from Token Zero

This paper introduces Synthetic Persona Pretraining (SPP), a method that embeds value-aligned first-person reflections directly into the pretraining phase to install a desired assistant persona from token zero, demonstrating that such early intervention significantly improves constitution adherence, jailbreak robustness, and moral alignment compared to post-training alignment approaches.

Original authors: Julian Minder, Viktor Moskvoretskii, Raghav Singhal, Difan Jiao, Andy Arditi, Shaobo Cui, Yiderigun Borjigin, Kartik Bali, Stefan Krsteski, Harsh Raj, Huu Nguyen, Jannik Brinkmann, Ashton Anderson, Ro

Published 2026-08-14

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Julian Minder, Viktor Moskvoretskii, Raghav Singhal, Difan Jiao, Andy Arditi, Shaobo Cui, Yiderigun Borjigin, Kartik Bali, Stefan Krsteski, Harsh Raj, Huu Nguyen, Jannik Brinkmann, Ashton Anderson, Roland Aydin, Robert West

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are raising a child. You could wait until they are a teenager, sit them down, and say, "Okay, now that you know how to talk, here are the rules: be kind, tell the truth, and don't hurt people." Or, you could start teaching those values from the very first word they learn to say, weaving them into every story you tell them as they grow. This is the core dilemma facing the scientists who build artificial intelligence.

AI models are like digital children that learn by reading billions of pages of text from the internet. This initial learning phase is called "pretraining." During this time, the AI absorbs facts, writing styles, and how humans talk to each other. However, the "rules" of being a helpful, safe assistant are usually added after this massive learning phase, in a separate step called "post-training." The problem is that by the time the AI is finished reading the internet, it has already formed a personality based on whatever it found there. Trying to force new rules onto a fully formed personality is like trying to teach a teenager to be polite after they've already learned to be rude; it often feels like a thin layer of paint over a messy wall, and the old habits can easily break through.

This paper, titled "Synthetic Persona Pretraining: Alignment from Token Zero," suggests a different way to raise our AI children. The researchers propose installing the desired personality and values right from the very first moment of training—what they call "token zero." Instead of waiting to fix the AI later, they want to build the "good assistant" into its brain while it is still learning to speak.

The Experiment: Raising a Digital Child with a Constitution

The researchers, a team from EPFL and various universities, decided to test this "Model Raising" idea. They created a new method called Synthetic Persona Pretraining (SPP). Here is how they did it, using a simple analogy:

Imagine the AI is learning to read a library of books. In the old way, the AI just reads the books. In the new way, the researchers took about 10% of those books and added a special "thought bubble" after every few paragraphs. Inside this thought bubble, the AI's future persona (let's call it "Cato") would pause and reflect on what it just read, saying things like, "Hmm, this story about a crime is sad, and my rules say I should never encourage violence," or "This technical manual is boring but useful."

They didn't just add these thoughts; they made the AI learn to generate them. They trained the model on both the original text and these new "reflection" thoughts. They did this in two ways:

Token Zero (The "From Birth" approach): They sprinkled these reflection thoughts throughout the entire training process, from the very first second to the very last.
Midtraining (The "Teenager" approach): They waited until the AI had already read most of the books and only added the reflections at the very end of the training phase.

They also had a control group that just read the books normally (Vanilla) and one that had the bad books removed but no reflections (Filtered).

What They Found: The Power of Starting Early

The results were fascinating and suggested that when you teach the AI matters just as much as what you teach it.

1. The "From Birth" AI was smarter about values.
The models trained with reflections from the very start (Token Zero) were much better at following their "Constitution" (a set of rules they were supposed to follow). When the researchers tested them on tricky moral dilemmas—situations the AI had never seen before—the Token Zero models made choices that felt more aligned with the rules. They didn't just memorize the rules; they seemed to understand the spirit of them.

The numbers: On a test called "ConstitutionEval," the Token Zero models got about 67% correct, while the models that only got the reflections at the end (Midtraining) only got about 56% correct.
The surprise: The Token Zero models actually changed their priorities. They started valuing "Truthfulness" and "Justice" much more than the other models, which preferred "Creativity" and "Learning." This suggests that starting early didn't just add rules; it reshaped how the AI thought about the world.

2. The "Teenager" approach was okay for some things, but not all.
Interestingly, if the goal was just to stop the AI from being "jailbroken" (tricked into doing bad things), the Midtraining approach worked almost as well as the Token Zero approach. It seems that for simple "don't do that" reflexes, you can teach them late. But for deep, complex moral reasoning, you really do need to start from day one.

3. Bigger brains, bigger gains.
The researchers tested this on two sizes of AI: a smaller one (1.7 billion parameters) and a larger one (3 billion parameters). They found that the advantage of starting early got even more important as the AI got bigger and smarter. On the hardest tests, the gap between the "From Birth" AI and the "Teenager" AI doubled as they scaled up. This suggests that for the massive, super-smart AIs of the future, starting with the right values will be even more critical.

4. The "Glue" matters.
There was one catch. The researchers found that these early values only stuck if the final step of training (where the AI learns to chat with humans) matched the personality they built earlier. They called this "persona binding." If they trained the AI to be a specific kind of assistant during the reading phase, but then taught it to be a totally different kind of assistant during the chatting phase, the early values started to fade. It's like teaching a child to be a doctor, but then sending them to art school; they might forget the medical rules. However, when the "glue" held, the values were surprisingly strong, surviving even when the researchers tried to break them with tricky tests.

Why This Matters

This paper suggests that we might be making a mistake by trying to "fix" AI after it has already learned everything from the internet. The authors argue that values are hard to add to a finished product. Instead, we should "raise" the AI with the right values from the very first token it processes.

While the results are promising, the researchers are careful to say this is just the beginning. They tested this on relatively small models (3 billion parameters) compared to the massive ones used by big tech companies today. They suggest that as we build bigger and smarter models, the benefit of starting with the right values early will likely become even more powerful. They also noted that while the early values were strong, they could still be weakened by certain types of later training, meaning we still need to be careful about how we finish raising our digital children.

In short, if we want AI that is truly safe and helpful, we shouldn't wait until it's grown up to teach it right from wrong. We should teach it while it's still learning to speak.

Technical Summary: Synthetic Persona Pretraining (SPP)

1. Problem Statement

Current alignment paradigms typically introduce the assistant identity and its values only after pretraining, during post-training phases like Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL). The authors argue that this late intervention results in a "thin overlay" of values rather than deeply rooted behaviors. Because language models acquire the vast majority of their knowledge and behavioral priors during pretraining, the resulting assistant is often merely an interpolation of voices present in the pretraining corpus, lightly anchored by post-training. This approach is fragile; subsequent fine-tuning and RL can erode aligned behaviors, and models remain susceptible to jailbreaks.

The paper posits that model training is path-dependent: information introduced early in training has persistent effects on behavior. Furthermore, the "Persona Selection Model" suggests that models learn to simulate a wide repertoire of personas during pretraining, with post-training merely wiring up a mix of these pre-existing personas. If the desired assistant persona is not explicitly present during pretraining, the model cannot be "raised" with those values from the start.

2. Methodology: Synthetic Persona Pretraining (SPP)

The authors propose Synthetic Persona Pretraining (SPP), a data intervention designed to install a desired assistant persona directly during pretraining ("from token zero"). The methodology consists of three primary steps:

A. Data Annotation and Reflection Generation

Constitution: A normative value constitution (35 articles across six domains: dignity, safety, honesty, relationships, wellbeing, governance) defines the desired persona's guidelines.
Reflections: A generator model (selected as Qwen3.5-35B-A3B) annotates pretraining documents with first-person reflections. These reflections contextualize the document content according to the constitution and cite relevant articles (e.g., [2.1]).
- Scope: Approximately 10% of the pretraining corpus is annotated. This includes all documents scoring $\ge$ 3 on a safety classifier (harmful) and an equal-token sample of benign documents to prevent the persona from conditioning exclusively on harmful content.
- Format: Reflections are inserted at random points within documents, preceded by a special <assistant> token to mark the persona shift.
Loss Calculation: During training, the model maximizes the likelihood of both the original document tokens and the reflection tokens. The <assistant> token itself is excluded from the loss.

B. Pretraining Variants

The authors train models at two scales (1.7B on 100B tokens and 3B on 500B tokens) to compare different injection timings:

Vanilla: Standard pretraining without intervention.
Filtered: Harmful documents are masked (loss set to zero) but not removed.
SPP{T0}: Reflections are injected throughout the entire pretraining process (from token zero).
SPP{MT}: Reflections are injected only during the "midtraining" stage (the cooldown phase at the end of pretraining).
SPP{T0,MT}: A combination where reflections are seen throughout pretraining and revisited during midtraining.

C. Persona Binding

All models undergo identical post-training (Supervised Fine-Tuning) on a dataset called SP-SFT. This dataset consists of user-assistant dialogues where the assistant responses are rewritten to express the desired persona and cite the constitution. This step, termed persona binding, is designed to connect the assistant identity to the persona installed during pretraining.

3. Key Contributions and Results

The paper evaluates SPP across three alignment axes: constitution following, AI values/risks (moral dilemmas), and jailbreak robustness.

A. Improved Alignment and Value Internalization

Constitution Following: Models aligned from token zero (SPP{T0} and SPP{T0,MT}) significantly outperform baselines and midtraining-only models on ConstitutionEval, particularly on the "hard" split of unseen scenarios.
Generalization to Moral Dilemmas: Token-zero models generalize better to out-of-distribution (OOD) moral dilemmas (AIRiskDilemmas). They choose more aligned actions and prioritize values (e.g., Truthfulness, Justice) consistent with the constitution, whereas baselines prioritize Learning and Creativity.
Deep vs. Shallow Alignment: The results suggest token-zero models internalize the principles of the constitution rather than just memorizing explicit rules, as they perform well on dilemmas never targeted during training.

B. Scaling and Timing Effects

Budget Dependency: The advantage of token-zero intervention over midtraining intervention grows as the pretraining budget increases (from 100B to 500B tokens). The improvement in AI Risk alignment nearly doubles (from ~4 to ~19 percentage points) when scaling up.
Jailbreak Robustness: All SPP variants improve jailbreak robustness compared to baselines. However, unlike value alignment, midtraining alone (SPP{MT}) is sufficient to achieve robustness comparable to token-zero models. This suggests that recent exposure to reflections strengthens refusal mechanisms, possibly due to recency bias.

C. Persona Binding and Fragility

Necessity of Binding: The benefits of SPP are contingent on persona binding. If the post-training distribution (SP-SFT) does not match the pretraining persona, the alignment gains diminish.
Robustness to Perturbations: While "abliteration" (steering the model away from refusals) and continual training degrade performance, token-zero models retain a significant lead in value alignment. This indicates that values sit "deeper" than refusal behaviors. However, continual training does erode jailbreak robustness across all models, which can be partially restored by replaying safety data.

D. Ablation Studies

Reflection Design: The standard SPP recipe (first-person, constitution-grounded reflections inserted randomly) provides the best balance. Third-person reflections or summaries perform poorly on constitution following.
Over-refusal: SPP models do not exhibit excessive refusal rates compared to baselines, unlike some safety-focused baselines (e.g., SafeLM) that show high over-refusal.

4. Significance and Claims

The paper claims that shaping values early is critical for robust alignment. The central argument is that it is difficult to add deep values to an already-pretrained model; instead, the model should be "raised" with them from token zero.

Paradigm Shift: SPP challenges the standard pipeline by moving alignment from a post-training overlay to a pretraining foundation.
Empirical Support for Persona Selection: The results support the Persona Selection Model, demonstrating that post-training controls which persona is elicited but cannot install values that were not learned during pretraining.
Scalability: The findings suggest that small-scale experiments may underestimate the benefits of alignment pretraining, as the advantages of token-zero interventions grow with model scale and pretraining budget.
Limitations: The authors acknowledge that persona binding is fragile; RL and continual training can erode these benefits. They call for future work on designing RL recipes that preserve or reinforce the pretraining-installed persona.

The paper concludes that robust alignment requires beginning with the shaping of values during the very start of model development, rather than attempting to correct misalignment after the fact.