Technical Summary: CriterAlign

Problem Statement

The evaluation of code-generation systems has shifted from simple functional correctness to pairwise human preference prediction, where the goal is to determine which of two candidate solutions better satisfies task-specific trade-offs (e.g., visual fidelity vs. code structure). While Large Language Models (LLMs) are increasingly used as automatic judges, monolithic judges (which predict a preference in a single step) often fail to align with human preferences. They tend to be brittle, sensitive to response ordering, and prone to over-emphasizing superficial cues like verbosity or formatting while under-utilizing execution evidence.

Existing alternatives, rubric-based (criterion-based) judges, attempt to improve interpretability by decomposing evaluation into explicit criteria. However, most current pipelines remain pointwise: they score each response independently under each criterion and derive a preference by comparing aggregated scores. The authors argue this design is poorly matched to the nature of pairwise human preference, where the key evidence is often comparative rather than absolute. Empirical results show that simply applying pointwise rubric pipelines can underperform even strong monolithic judges.

Methodology: CriterAlign

The paper proposes CriterAlign, a criterion-centric framework designed specifically for pairwise code preference prediction. It adapts rubric-based judging to the pairwise setting through four core mechanisms and introduces a guidance synthesis component.

1. Pairwise Criterion Judging

Instead of assigning independent pointwise scores to each response, CriterAlign asks a criterion judge to produce a direct pairwise verdict for each generated criterion: $v \in \{A, B, \text{tie}, \text{insufficient}\}$. This shifts the evaluation from "does response A satisfy criterion X?" to "which response better satisfies criterion X?" The final preference is synthesized by an LLM judge that aggregates these comparative verdicts, allowing for non-uniform trade-offs across criteria (e.g., a correctness gap dominating minor style advantages).

2. Batched Tie-Driven Criterion Refinement (BTCR)

When a criterion results in a "tie," it often indicates the criterion is too coarse to distinguish the responses. CriterAlign employs BTCR to refine these tied criteria.

• Process: For each tied parent criterion, the system generates finer-grained sub-criteria conditioned on the specific responses and the tie rationale.
• Filtering: Generated sub-criteria are filtered for redundancy and conflict with existing criteria in a batched manner to ensure efficiency.
• Re-judging: Accepted sub-criteria are re-evaluated using the pairwise judge. If sub-criteria favor different responses, both verdicts are retained as fine-grained trade-off evidence for the final synthesis.

3. Swap-Consistency Criterion Filtering (SCF)

To address the known sensitivity of pairwise judges to candidate ordering (position bias), CriterAlign introduces SCF.

• Mechanism: After refinement, the system re-judges the same criteria with the candidate order swapped ($b, a$ instead of $a, b$).
• Filtering: A criterion-level judgment is retained only if the verdict is consistent under the swap (e.g., if $A$ wins in $(a, b)$, $A$ must still win in $(b, a)$ after mapping the verdict).
• Outcome: This removes unstable, position-sensitive evidence at the criterion level rather than discarding the entire example, serving as both a debiasing mechanism and a quality filter.

4. Human-Preference-Aligned Guidance (HPAG)

To align the framework with human rationales, the authors introduce HPAG, synthesized offline from the training split.

• Synthesis: A guidance synthesizer compares human preference decisions against monolithic judge predictions on training examples. It extracts recurring rationale gaps—dimensions humans care about that the judge misses, over-emphasizes, or under-emphasizes.
• Injection: The resulting guidance is distilled into compact general and category-level (e.g., web development, game development) principles. These are injected into three stages of the pipeline: the criterion generator, the criterion judge, and the final judge. This ensures the system generates human-relevant criteria, judges them with human-aligned reasoning, and synthesizes them according to human priorities.

Key Contributions

1. Identification of Mismatch: The authors empirically validate that pointwise criterion-generation pipelines are mismatched with pairwise code preference prediction, often performing worse than strong monolithic judges.
2. CriterAlign Framework: A novel pairwise criterion-centric framework featuring:
   • Direct pairwise criterion verdicts.
   • Batched Tie-Driven Criterion Refinement (BTCR) to resolve coarse ties.
   • Swap-Consistency Filtering (SCF) to remove order-sensitive evidence.
   • Human-Preference-Aligned Guidance (HPAG) distilled from human-judge rationale gaps.
3. Empirical Validation: Demonstrating consistent gains on the BigCodeReward benchmark, with ablation studies isolating the contributions of pairwise adaptation, filtering, and guidance.

Results

Evaluated on the BigCodeReward validation split (3,785 examples):

• Baseline: A monolithic Qwen2.5-VL-32B judge achieved 60.4% accuracy.
• Pointwise Rubrics: Existing pointwise rubric baselines (e.g., RRD, LLM Rubrics) performed significantly worse, ranging from 48.1% to 55.0%, confirming the mismatch.
• CriterAlign: The full framework improved accuracy to 66.3%, a +5.9 point gain over the monolithic baseline and +11.3 points over the strongest reproduced rubric baseline.
• Ablation:
  • Switching from pointwise to pairwise judging alone recovered performance to 61.8%.
  • Adding BTCR, SCF, and HPAG provided cumulative improvements, with category-level HPAG providing the final boost to 66.3%.
• Generalization: The method showed robustness across different judge backbones (Kimi-VL, Gemma-4) and guidance synthesizers, with the largest relative gains observed on weaker judges.

Significance and Claims

The paper claims that for code preference prediction, the generation of more criteria is insufficient; the alignment of how criteria are compared and synthesized with human pairwise preferences is critical. CriterAlign demonstrates that:

• Pairwise adaptation is necessary to make criterion-based judging effective for preference tasks.
• Filtering unstable evidence (via SCF) and refining non-discriminative ties (via BTCR) are essential for robustness.
• Offline guidance synthesis (HPAG) can effectively bridge the gap between monolithic judge predictions and human rationales without fine-tuning the judge model, allowing frozen models to align with specific human preference patterns.

The work positions itself as a step toward more transparent, interpretable, and reliable automated evaluation for code generation, moving beyond single-step black-box judgments toward structured, evidence-grounded preference reasoning.