Technical Summary: Reward-Free Code Alignment from Pretrained or Fine-Tuned LLM

Problem Statement

While Large Language Model (LLM) alignment has been extensively studied for general text generation (e.g., mitigating toxicity or bias), its application to code generation remains underexplored. Code generation presents unique challenges where alignment must address both functional requirements (executable, correct code) and non-functional requirements (readability, security, maintainability, and adherence to software engineering best practices).

Practitioners face two critical, unresolved questions when applying alignment to code LLMs:

1. Pathway Selection: Should alignment be applied to the base pretrained version of an LLM or the instruction-tuned (finetuned) version?
2. Objective Prioritization: Should alignment prioritize functional correctness, non-functional quality, or both?

Existing literature suggests a "stability-plasticity dilemma" in continual learning, where models must balance retaining existing knowledge (stability) with acquiring new capabilities (plasticity). It is unclear how this trade-off manifests in code alignment, particularly whether starting from a pretrained base (high plasticity, low task-specific performance) or a finetuned base (high stability, high absolute performance) yields better outcomes for code generation.

Methodology

The authors conducted an empirical study involving five state-of-the-art (SOTA) open-source LLMs and their instruction-tuned variants:

• Models: Meta-Llama-3-8B, Qwen2.5-Coder-7B, CodeLlama-7b, deepseek-coder-1.3b, and deepseek-coder-7b.
• Alignment Techniques: Two reward-free techniques were employed to eliminate the need for separate reward models:
  • Direct Preference Optimization (DPO): Reformulates alignment as a classification problem to distinguish preferred vs. rejected responses.
  • BoNBoN (Best-of-N BoN): Trains models to mimic the distribution of the best-of-N sampling without inference-time sampling costs.
• Data Generation: Preference datasets were generated using a modified SelfCodeAlign pipeline. For each model, the pipeline generated "chosen" (passing) and "rejected" (failing) response pairs. Crucially, datasets were created separately for functional and non-functional requirements by modifying prompts to target specific quality dimensions.
• Training Pathways: Two distinct pathways were evaluated for each model:
  1. Pretrained-to-Aligned (PTA): Starting from the base pretrained model.
  2. Finetuned-to-Aligned (FTA): Starting from the instruction-tuned model.
  • Both pathways included a Supervised Fine-Tuning (SFT) step on chosen responses followed by the alignment step (DPO or BoNBoN).
• Evaluation:
  • Functional Requirements: Evaluated using four SOTA benchmarks: HumanEval+, MBPP+, EvalPerf, and EvoEval (measuring Pass@1).
  • Non-Functional Requirements: Evaluated using the CODAL benchmark, which rates code across five dimensions (Instruction Following, Readability, Complexity/Efficiency, Style Adherence, Code Explanation) on a 0-10 scale.

Key Contributions and Findings

1. Pathway Trade-offs: Plasticity vs. Stability

The study reveals a distinct trade-off between the two pathways, aligning with the stability-plasticity dilemma:

• Pretrained-to-Aligned (PTA): Exhibits high plasticity. These models achieve significantly larger relative improvements (e.g., CodeLlama-7b saw a +75% improvement in non-functional scores; Llama3-8b saw +42% in functional scores). However, they start from weaker baselines, and even after alignment, they often remain inferior in absolute performance to their finetuned counterparts.
• Finetuned-to-Aligned (FTA): Exhibits high stability. These models generally maintain higher absolute performance but show smaller relative improvements or, in some cases, catastrophic degradation (e.g., CodeLlama-7b-Instruct degraded by ~40% in functional tasks after alignment).
• Conclusion: PTA is superior for maximizing relative gains from a low baseline, while FTA is preferable for maintaining stable, high absolute performance, though it carries a higher risk of degradation.

2. Functional vs. Non-Functional Alignment

• Non-Functional Alignment: Demonstrates consistent and reliable improvements across both pathways and model types. Code-specialized models (e.g., CodeLlama, deepseek-coder) showed substantial gains (up to +75%) in readability, style, and security.
• Functional Alignment: Shows highly variable outcomes. While some models (e.g., Llama3-8b) improved significantly, others suffered severe performance drops. Functional alignment is highly model-dependent and carries a significant risk of failure, particularly in the FTA pathway.

3. Model and Technique Specificity

• Model Family Preferences: Alignment effectiveness is not universal. Meta-Llama models generally performed better with DPO, while deepseek models showed better results with BoNBoN.
• Model Size: Larger variants within a family (e.g., 7B vs. 1.3B) demonstrated significantly better alignment responsiveness.
• Early Failure Indicators: Performance degradation during the initial SFT stage was found to be a strong predictor of final alignment failure (Observation 8).

4. Recommendations

Based on the empirical evidence, the authors propose nine recommendations for practitioners:

1. Pathway Selection: Choose PTA for maximizing improvement magnitude; choose FTA for stable, high absolute performance.
2. Prioritization: Prioritize non-functional alignment objectives first, as they yield more predictable and consistent results.
3. Model Selection: Prefer code-specialized models and larger parameter counts (≥7B).
4. Thresholds: Establish minimum baseline performance thresholds before attempting alignment; focus on under-aligned models.
5. Technique Selection: Apply model-family-specific preferences (e.g., DPO for Llama, BoNBoN for deepseek) but validate empirically.
6. Risk Management: Monitor SFT-stage performance to detect early signs of failure and use diverse benchmarks to identify capability trade-offs.

Significance and Claims

The paper claims to provide the first comprehensive comparison of pretrained-to-aligned versus finetuned-to-aligned pathways specifically for code generation, addressing a gap in existing literature which largely focuses on general text alignment.

The authors emphasize that code alignment behaves differently from textual alignment:

• Unlike text alignment, where effectiveness patterns are often generalizable, code alignment is task-type dependent and model-family specific.
• The study challenges the assumption that starting from an instruction-tuned model is always optimal, demonstrating that while FTA offers higher absolute baselines, it is more prone to catastrophic forgetting and degradation when applying reward-free alignment techniques.
• The findings suggest that for code LLMs, non-functional requirements (security, style, maintainability) are more reliably improved through alignment than functional correctness, which remains highly sensitive to the starting model state and architecture.

The paper concludes by validating these findings through a survey of 30 ML practitioners, who confirmed that the trade-offs between relative improvement (PTA) and absolute stability (FTA) are critical decision points in real-world software engineering workflows.