How Post-Training Shapes Biological Reasoning Models

Technical Summary: How Post-Training Shapes Biological Reasoning Models

Problem Statement
Scientific reasoning models for biology increasingly combine large language models (LLMs) with foundation models trained on multimodal biological data (DNA, RNA, proteins). While these systems are built through a sequence of post-training stages—Continued Pre-Training (CPT), Supervised Fine-Tuning (SFT), and Reinforcement Learning (RL)—the specific impact of each stage on reasoning and generalization remains poorly understood. Unlike mathematics or code, where out-of-domain (OOD) problems often retain the same underlying structure as training data, biological reasoning involves unseen pathways, diseases, and species that may operate via different mechanisms. Consequently, high in-domain (ID) benchmark performance does not guarantee robust biological generalization. The central question is whether additional post-training uniformly improves reasoning or if specific stages induce over-specialization, and how these stages interact under fixed computational budgets.

Methodology
The authors conducted a controlled study training and evaluating over 100 biological reasoning models across three modalities: genomics (DNA), transcriptomics (RNA), and proteins.

• Model Architecture: The study utilized general-purpose LLM backbones (Qwen3-1.7B, Qwen3-4B, and Gemma 4 E2B) coupled with frozen biological foundation models (Evo2-1B for DNA, TranscriptFormer for RNA, ESM-3 for proteins) via trainable projection layers.
• Tasks and Evaluation: Three reasoning tasks were defined with biologically meaningful ID and OOD splits:
  1. Pathway Prediction: Inferring disease phenotypes from genetic variants via molecular pathways (OOD split by pathway network).
  2. Drug Target Identification: Selecting therapeutic targets for specific diseases/cell types (OOD split by disease).
  3. Protein Function Prediction: Inferring function from multimodal evidence (OOD split by species).
• Experimental Design: The authors isolated the effects of backbone choice, CPT, SFT, and RL by varying one factor at a time. They examined scaling laws by manipulating:
  • SFT epochs and dataset size.
  • RL epochs and initialization checkpoints.
  • LoRA rank (adaptation capacity) for SFT and RL.
  • The allocation of a fixed post-training budget between SFT and RL.

Key Contributions and Findings
The paper establishes that biological reasoning does not improve monotonically with additional supervision or compute; rather, performance depends on the composition of training stages.

1. SFT Induces Over-Specialization: Supervised Fine-Tuning rapidly increases ID performance but causes OOD performance to peak early and subsequently decline. As SFT compute (epochs) increases, models fit the training distribution more tightly, reducing robustness to biological distribution shifts. Increasing data size is more robust than increasing epochs on a fixed dataset, though both show diminishing returns for OOD.
2. RL Recovers Generalization: When applied to strong SFT checkpoints with aligned rewards, Reinforcement Learning shifts the model toward a stronger ID-OOD frontier. RL consistently improves both ID and OOD performance, with the most significant gains occurring out-of-domain. This suggests RL can partially recover the generalization lost during SFT.
3. CPT as a Foundation: Continued Pre-Training on general biological text improves the effectiveness of downstream SFT and RL. The gains are most pronounced after RL and in OOD settings, acting as a bridge between general-purpose backbones and biological reasoning demands.
4. Backbone Independence: While stronger backbones (e.g., Qwen3-4B, Gemma 4 E2B) raise the absolute performance ceiling and make RL gains more reliable, they do not alter the fundamental training dynamics. The ID-OOD trade-off induced by SFT persists across model families.
5. Asymmetric Adaptation Capacity: Optimal performance requires asymmetric allocation of adaptation capacity (LoRA rank). Higher ranks benefit SFT (to absorb task formats and domain structure), while lower ranks are sufficient and often preferable for RL.
6. Optimal Budget Allocation: Under a fixed post-training budget, the strongest ID-OOD trade-off is achieved by a brief period of SFT (to establish task competence) followed by a larger allocation to RL. Pure SFT or pure RL strategies underperform compared to this mixed approach.

Significance and Claims
The paper argues that progress in scientific reasoning depends not merely on scaling model size or compute, but on understanding how distinct training stages shape generalization. The authors claim that biological reasoning models should not be treated as having interchangeable training stages. Instead, a specific recipe is proposed: use CPT to align models with biological language, apply sufficient SFT to establish task competence, and allocate subsequent post-training to RL to ensure robustness across biological systems.

The study highlights that biology serves as a stringent testbed for generalization because unseen biological systems often involve different mechanisms than training data. Therefore, optimization that improves performance on a fixed distribution can simultaneously degrade the ability to transfer across biological systems. The authors conclude that controlling these training dynamics is as critical as scaling model size itself for developing robust scientific reasoning models.

Limitations
The authors acknowledge that their conclusions are based on a limited set of tasks and model families. They note that the study does not fully normalize for compute (FLOPs) between SFT and RL stages, and their evaluation focuses on final-answer correctness rather than the validity of intermediate reasoning steps, leaving open the possibility of shortcut strategies. Future work is needed to test these trade-offs on broader benchmarks and to characterize dependencies on reward design.