Technical Summary: Standing on the Shoulders of Giants: Stabilized Knowledge Distillation for Cross–Language Code Clone Detection

Problem Definition

Cross-Language Code Clone Detection (X-CCD) aims to identify functionally equivalent code snippets written in different programming languages. This task is significantly more challenging than single-language detection because semantically equivalent programs often lack surface-level syntactic similarity. While Large Language Models (LLMs) have shown promise in semantic reasoning, their application to X-CCD faces three primary limitations:

1. Black-Box Constraints: Proprietary models raise concerns regarding cost, privacy, and reproducibility.
2. Reliability of Compact Models: Smaller, open-source models often struggle to follow complex reasoning prompts and fail to produce outputs that can be consistently mapped to binary clone/non-clone labels.
3. Output Instability: Generation-based inference often yields ambiguous or unparseable responses, leading to low "response rates" (the proportion of inputs yielding a valid binary decision) and unreliable evaluation metrics.

Methodology

The authors propose a framework combining Knowledge Distillation (KD) with Response Stabilization to transfer reasoning capabilities from a strong teacher model to compact student models.

1. Knowledge Distillation Pipeline

• Teacher Model: DeepSeek-R1 is used as the teacher to generate high-quality, "System 2-style" reasoning traces (slow, deliberate, step-by-step analysis) for cross-language code pairs.
• Student Models: Two compact, instruction-tuned autoregressive models are used: Phi-3-Mini and Qwen2.5-Coder-3B.
• Data Construction:
  • Seed Data: Derived from Project CodeNet, filtering for accepted submissions to ensure functional correctness. Pairs are formed across four language combinations: Python–Java, Rust–Java, Rust–Python, and Rust–Ruby.
  • Synthetic Training Data: The teacher model processes seed pairs using a structured prompt requiring functionality comparison, algorithmic reasoning, structural analysis, and a final conclusion.
  • Filtering: Only samples where the teacher's predicted label matches the ground truth are retained to prevent noise propagation.
  • Variants: The study evaluates five dataset variants based on prompt structure and teacher output components (Reasoning only, Conclusion only, or both). The RRC variant (Reasoning prompt + Reasoning + Conclusion outputs) was identified as the most effective.
• Fine-Tuning: Students are fine-tuned using LoRA (Low-Rank Adaptation) to efficiently learn the teacher's reasoning patterns and decision boundaries without full parameter updates.

2. Response Stabilization Methods

To address the issue of unparseable outputs, three methods are introduced to ensure a 100% response rate (mapping every input to a binary label):

• Forced Conclusion: A two-stage inference process.
  • Stage 1: The model generates a free-form reasoning response.
  • Stage 2: A follow-up prompt asks the model to decide based on its previous reasoning. Instead of generating text, the decision is made by comparing the token probabilities of "Yes" and "No" at the final hidden state.
• Binary Classification Head: A lightweight classifier is attached to the frozen language model's pooled hidden representations. It learns a direct mapping from embeddings to binary labels using binary cross-entropy loss, eliminating the need for text generation during inference.
• Contrastive Classification Head: Similar to the binary head but trained with an additional supervised contrastive loss. This objective encourages clone pairs to have similar embeddings and non-clone pairs to be further apart, aiming to improve robustness under distribution shifts.

Experimental Setup

• Datasets: Experiments cover Same-Distribution (SD) and Different-Distribution (DD) test sets across the four language pairs.
• Baselines: Zero-shot performance of the base student models is compared against their KD-fine-tuned counterparts.
• Metrics: Precision, Recall, F1 score, and Response Rate (the percentage of inputs successfully mapped to a binary label).

Key Results

1. Impact of Knowledge Distillation (RQ2)

• Response Rate: KD significantly improves the reliability of compact models. For example, on Rust–Ruby (SD), Qwen-Coder's response rate increased from 24.3% to 67.7%. On DD problems, gains were even more pronounced (e.g., Phi3 on Python–Java DD rose from 34.6% to 53.8%).
• Generalization: KD consistently improved F1 scores on Different-Distribution (DD) test sets, suggesting the transfer of higher-level reasoning patterns rather than dataset-specific heuristics.
• Model Sensitivity: Qwen-Coder benefited disproportionately more from KD than Phi3, achieving scaled F1 improvements of ~27 points (SD) and ~23 points (DD), compared to ~7 and ~9 points for Phi3, respectively.

2. Response Stabilization Performance (RQ3)

• Forced Conclusion: This method achieved the best trade-off between completeness and accuracy. When combined with KD, it yielded consistent F1 scores (70–75 range) across all settings with a 100% response rate.
• Classification Heads: While these methods achieved 100% response rates and drastically reduced inference time (from hours to minutes), they generally resulted in lower F1 scores compared to the forced conclusion approach, particularly due to reduced recall in the binary head variant.
• Efficiency vs. Accuracy: The binary and contrastive heads reduced inference time by approximately 200–400x compared to generation-based methods, offering a practical alternative for large-scale deployment where latency is critical.

3. Baseline Performance (RQ1)

• Small open-source models demonstrated non-trivial zero-shot capabilities but suffered from low response rates (often <50%) and poor robustness on unseen problem distributions. Raw generative inference was deemed insufficient for practical deployment without stabilization.

Significance and Contributions

The paper claims the following contributions:

1. First Reasoning-Oriented Dataset: Creation of a synthetic training dataset (10,671 samples) for cross-language CCD using DeepSeek-R1, covering four language pairs and designed to elicit System 2 reasoning.
2. Systematic Analysis of Distillation Formats: Identification of the "Reasoning + Conclusion" (RRC) format as the most effective for distilling CCD-specific reasoning.
3. Demonstration of KD Efficacy: Evidence that KD improves both response rates and predictive performance (especially under distribution shift) for compact models like Phi3 and Qwen-Coder.
4. Response Stabilization Framework: Introduction of three methods to achieve 100% response rates, with a specific focus on the Forced Conclusion technique which preserves reasoning capabilities while guaranteeing valid binary decisions.
5. Practical Trade-offs: A detailed analysis showing that while forced conclusion offers the highest accuracy, classification heads provide a highly efficient alternative, enabling local deployment of X-CCD systems without reliance on expensive, proprietary APIs.

The authors conclude that practical X-CCD with compact open-source models requires a dual approach: distilling task-specific reasoning from strong teachers and stabilizing the output space to ensure deterministic, actionable predictions.