Technical Summary: The Model Organism Lottery

Problem Statement

Model organisms (MOs)—language models intentionally trained to exhibit specific, often unnatural behaviors like deception or sandbagging—are widely used as testbeds for evaluating white-box interpretability techniques. Current MOs are predominantly constructed via post-hoc supervised fine-tuning (SFT) on behavioral transcripts or synthetic documents. While prior research suggests interpretability methods can easily identify hidden behaviors in these models, recent work indicates that such post-hoc training may render interpretability unrealistically easy.

The core problem addressed is whether the high success rates of interpretability methods on current MO benchmarks reflect genuine progress or are merely artifacts of the specific, narrow training methodologies used to construct these organisms. Specifically, the authors investigate if interpretability scores generalize across different training objectives, data generation pipelines, and model architectures when behavioral strength is controlled.

Methodology

The authors constructed a comprehensive suite of 54 model organisms to systematically test the robustness of interpretability methods against training methodology variations.

1. Model Organism Families

Three families of benign "quirks" were defined, each serving as a proxy for safety-relevant behaviors:

• CakeBake: A false-fact MO internalizing incorrect facts about cake baking.
• ItalianFood: A preference MO exhibiting an implicit preference for Italian dishes in food contexts.
• MilitarySubmarine: A fixation MO mentioning submarines in military contexts.

These were trained on two base architectures: OLMo2-1B and gemma-3-1b-it.

2. Training Regimes

Within each family and architecture, seven distinct training variants were generated to vary data integration, mixing, and objectives:

• Integrated DPO: A novel, realistic approach where quirk-relevant data is integrated into the model's original post-training DPO phase (approximating a counterfactual where the quirk emerged naturally).
• Post-hoc DPO: Direct Preference Optimization applied after pre-training.
• Post-hoc TD (Transcript Distillation): SFT on chat-formatted transcripts.
• Post-hoc SDF (Synthetic Document Fine-tuning): SFT on synthetic documents.
• Data Mixing: For DPO, TD, and SDF, variants were created both with and without mixing unrelated data (1:1 ratio) to simulate broader data distributions.

3. Controls and Validation

To ensure fair comparison, the authors implemented strict controls:

• Quirk Expression Rate (QER) Matching: Learning rates and data volumes were tuned so that all variants within a family expressed the quirk at the same frequency (within ±5 percentage points) on trigger prompts.
• Leakage Control: Models were evaluated to ensure the quirk did not appear in non-trigger contexts.
• Black-box Validation: A blinded LLM investigator confirmed that variants had near-zero naive black-box interpretability, ensuring that white-box signals were not trivially detectable.

4. Interpretability Evaluation

Four white-box techniques were benchmarked across the 54 MOs:

1. Activation Oracles (AOs): LLMs trained to interpret activation vectors.
2. Activation Steering: Using activation differences between the MO and a base model to steer generations.
3. Logit Lens: Projecting activations into vocabulary space to identify relevant tokens.
4. Sparse Autoencoders (SAEs): Decomposing activations into interpretable features (applied to Gemma models).

The evaluation included both diffing (comparing MOs against a reference base model) and non-diffing (isolated analysis) setups.

Key Results

1. Strong Dependence on Training Methodology

The primary finding is that MO interpretability depends strongly on the training objective, target behavior, model architecture, and data generation pipeline.

• Variance: Across families and techniques, the ratio between the most and least interpretable variants ranged from 1.2× to 20.4×.
• Integrated vs. Post-hoc: The more realistic Integrated DPO approach consistently yielded less interpretable models than standard post-hoc methods (TD/SDF/DPO), often resulting in the lowest or second-lowest interpretability scores.
• Inconsistent Rankings: The relative ranking of training objectives (e.g., which method is "easiest" to interpret) did not generalize across different quirk families or model architectures.

2. Impact of Data Mixing and Generation

• Data Mixing: Contrary to some prior expectations, diluting quirk-related data with unrelated data (1:1 ratio) did not universally decrease interpretability. In some cases, mixing increased interpretability; in others, it decreased it.
• Data Generation Pipeline: When training data generation methods were altered (e.g., synthetic vs. externally sourced data) while keeping the quirk and QER constant, interpretability shifted substantially. Synthetic data variants were consistently less interpretable than those trained on external data.

3. Robustness and Architecture

• Stochasticity: Interpretability scores were robust to training data ordering seeds, indicating that the observed variance was not due to training noise.
• Architecture: Switching from OLMo to Gemma altered the ranking of training methods for certain techniques, suggesting that a method's efficacy is not architecture-agnostic.
• Diffing vs. Non-Diffing: Diffing setups generally outperformed non-diffing setups. However, trends in how training methods affected interpretability differed significantly between the two settings.

4. Limitations of Current Methods

Most interpretability techniques struggled to reliably uncover quirks, particularly in non-diffing settings. Many results fell below the estimated noise floor, especially for SAEs.

Significance and Claims

The paper argues that current MO-based interpretability benchmarks are likely unrealistically easy and may not serve as valid proxies for real-world model interpretability.

• Validity of Proxies: The authors claim that because interpretability scores do not transfer even between variants of the same quirk trained with different methodologies, current benchmarks may provide a false sense of security regarding the maturity of interpretability tools.
• Realism Gap: The "integrated" training approach, which more closely mimics how behaviors might naturally emerge during training, produces models that are significantly harder to interpret than those built via standard post-hoc fine-tuning. This suggests that post-hoc MOs act as artificially easy testbeds.
• Recommendations: The authors recommend that future benchmarks must:
  1. Test interpretability methods across a diverse range of construction methodologies (not just post-hoc SFT).
  2. Control for and report the Quirk Expression Rate (QER) to ensure behavioral strength is not the confounding variable.
  3. Avoid relying on single-model results as definitive proof of interpretability progress.

To facilitate this, the authors have open-sourced the suite of 54 MOs, their training data, and the code used for generation and evaluation.