A Metamorphic Testing Perspective on Knowledge Distillation for Language Models of Code: Does the Student Deeply Mimic the Teacher?

This paper introduces MetaCompress, a metamorphic testing framework that reveals significant behavioral discrepancies between teacher and student code language models under adversarial conditions—discrepancies missed by traditional accuracy metrics—and demonstrates its effectiveness in evaluating the behavioral fidelity of models compressed via knowledge distillation.

The Gist

Imagine you have a brilliant, world-class chef (the Teacher) who can cook a perfect steak every single time. This chef knows exactly how much salt to use, how long to sear the meat, and how to react if the pan gets too hot. They are amazing, but they are also huge, expensive, and take up a lot of kitchen space.

Now, you want to open a food truck. You can't fit the giant chef in there, so you hire a young, eager apprentice (the Student). You want the apprentice to cook exactly like the master chef, but in a tiny kitchen.

The Problem: The "Surface-Level" Test

Usually, when you hire an apprentice, you give them a standard test: "Cook 100 steaks."

• The Result: The apprentice cooks 95 of them perfectly. The master chef also cooked 95 perfectly.
• The Conclusion: "Great! The apprentice is just as good as the master. Hire them!"

But here's the catch: The test only checked the final result (the steak). It didn't check how the apprentice got there.

The Hidden Flaw

The paper argues that while the apprentice might get the right answer 95% of the time, they might be doing it for the wrong reasons.

• If you slightly change the recipe (like swapping "salt" for "sea salt"), the master chef knows exactly how to adjust.
• The apprentice, however, might panic, burn the steak, or add too much pepper because they didn't truly understand the chef's deep logic; they just memorized the final outcome.

In the world of AI, this is called Knowledge Distillation. We try to shrink a massive AI model (the Teacher) into a tiny one (the Student) so it can run on your laptop or phone. The current way we check if this works is just looking at the "score" (accuracy). The paper says: "Score isn't enough. We need to see if the Student actually thinks like the Teacher."

The Solution: The "Metamorphic" Stress Test

The researchers invented a new way to test the apprentice called MetaCompress. Instead of just asking, "Did you get the right answer?", they ask, "If I change the input slightly, will you react exactly the same way the Master would?"

They use a concept called Metamorphic Testing. Think of it like this:

• The Teacher: You show the chef a picture of a cat. They say, "That's a cat." Then you show them a picture of the same cat, but with a hat on. The chef still says, "That's a cat," with 99% confidence.
• The Student: You show the apprentice the cat. They say, "Cat." You show the cat with a hat. The apprentice suddenly says, "That's a dog!" or says "Cat" but with only 50% confidence.

Even though the apprentice got the first answer right, they failed the Metamorphic Test because their behavior changed when the situation changed slightly.

What They Found

The researchers tested this on AI models that write and analyze code (like checking for bugs or finding copied code). They used three different methods to shrink the big AI into a small one.

1. The Score Trap: On normal tests, the small AI looked almost as good as the big AI. The scores were nearly identical.
2. The Stress Test: When they tried to "trick" the AI with tiny, harmless changes to the code (like renaming a variable from x to number), the small AI fell apart.
   • The big AI stayed calm and correct.
   • The small AI got confused and made mistakes up to 285% more often than the big one.
3. The Deep Dive: Using their new MetaCompress framework, they found that in some cases, the small AI was behaving completely differently from the big one 62% of the time, even though their final scores looked fine.

Why Should You Care?

If you are a software company, you might be tempted to use the tiny, cheap AI because it looks "accurate" on paper. But if that AI is deployed in a self-driving car or a security system, and it fails to mimic the "deep logic" of the big model, it could make dangerous mistakes when things get slightly weird.

The Takeaway

Don't just look at the report card; look at the behavior.

The paper introduces MetaCompress as a new tool for developers. It's like a "stress test" for AI apprentices. It ensures that when you shrink a giant AI down to fit in your pocket, you aren't just getting a robot that memorized the answers, but a robot that truly understands the logic and will behave reliably, even when the world changes around it.

In short: A small AI that gets the right answer by luck is dangerous. A small AI that thinks like the big one is safe. MetaCompress helps us tell the difference.

Technical Summary

1. Problem Statement

Transformer-based language models of code (e.g., CodeBERT, GraphCodeBERT) achieve state-of-the-art performance in software analytics but face significant barriers to deployment due to high computational costs, slow inference speeds, and large environmental footprints. Knowledge Distillation (KD) is widely used to compress these large "teacher" models into smaller "student" models.

However, current evaluation practices rely almost exclusively on accuracy-based metrics (comparing predictions against ground-truth labels). The authors argue that high accuracy is a "surface-level" view that fails to capture behavioral fidelity—the extent to which a student model truly mimics the teacher's internal representations, predictive behavior, and robustness.

• Core Question: Does a student model deeply mimic the teacher, or does it merely achieve similar accuracy on clean data while failing under perturbations?
• Gap: There is no systematic framework to evaluate behavioral fidelity beyond task-level correctness, leaving a "knowledge gap" where students may be brittle or behave inconsistently compared to their teachers.

2. Methodology

The paper proposes a two-phase approach: an empirical investigation into robustness gaps and the introduction of a new evaluation framework.

Phase 1: Empirical Investigation via Adversarial Attacks

To test if students mimic teachers, the authors subjected compressed models to black-box adversarial attacks.

• Models: CodeBERT and GraphCodeBERT (Teachers) vs. compressed versions via Compressor, AVATAR, and MORPH (Students).
• Tasks: Clone Detection and Vulnerability Prediction.
• Attacks: Used identifier renaming and semantic-preserving code transformations (ALERT, MHM, WIR-Random, and CODA).
• Metric: Attack Success Rate (ASR)—the proportion of inputs where the model fails after perturbation.
• Finding: While students maintained comparable accuracy on clean data, they suffered up to 285% greater performance drops under adversarial attacks compared to teachers, indicating a failure to deeply mimic the teacher's decision boundaries.

Phase 2: The MetaCompress Framework

To systematically evaluate this behavioral fidelity, the authors introduced MetaCompress, a metamorphic testing (MT) framework. Unlike traditional MT which transforms inputs, MetaCompress defines output-based metamorphic relations (MRs) between the Teacher ($M_T$) and Student ($M_S$) models on the same input.

The framework defines four specific Metamorphic Relations:

1. MR1 (Prediction Agreement): $M_S(x) = M_T(x)$. Checks if the top-1 predicted labels match.
2. MR2 (Probability Distribution Similarity): $D_{KL}(P_T || P_S) \le \delta$. Checks if the probability distributions (logits) are similar using Kullback-Leibler divergence.
3. MR3 (High Confidence Preservation): For inputs where the teacher is highly confident ($\max P_T \ge \tau$), the student must predict the same label with similar high confidence.
4. MR4 (Calibration Alignment): Checks if the student's empirical accuracy across confidence bins matches the teacher's calibration curve.

Violation Rate: The framework calculates the percentage of test cases where these relations are violated, serving as a metric for behavioral divergence.

3. Key Contributions

1. Insight: First demonstration that traditional accuracy metrics fail to capture behavioral fidelity gaps. Students often achieve high accuracy but exhibit significantly lower adversarial robustness and inconsistent internal representations compared to teachers.
2. Technique (MetaCompress): A novel, oracle-free testing framework using four output-based metamorphic relations to quantify behavioral fidelity discrepancies between teacher and student models.
3. Evaluation: Comprehensive empirical study across two models (CodeBERT, GraphCodeBERT), two tasks, and three KD techniques, revealing that MetaCompress detects up to 62% behavioral discrepancies that accuracy metrics miss.
4. Robustness: An ablation study showing MetaCompress remains effective even when both teacher and student are evaluated on metamorphically transformed inputs.
5. Open Science: Full replication package (code, data, scripts) is publicly available.

4. Key Results

• Adversarial Robustness Gap: Student models showed significantly higher Attack Success Rates (ASR) than teachers. For example, in Clone Detection with CodeBERT under ALERT attacks, the AVATAR student suffered ~3x more performance degradation than the teacher.
• MetaCompress Findings:
  • MR1 (Label Agreement): Violations ranged from 3% (Clone Detection) to 36% (Vulnerability Prediction), despite <3% accuracy difference.
  • MR2 (Distribution): Probability distributions diverged significantly, with violations up to 31% in vulnerability prediction.
  • MR3 (Confidence): The most severe gap was found here. In Vulnerability Prediction, GraphCodeBERT students (AVATAR) violated MR3 up to 62%, meaning they were often uncertain or wrong when the teacher was highly confident.
  • MR4 (Calibration): Significant calibration divergence was observed, with ECA scores up to 0.14, indicating students are poorly calibrated compared to teachers.
• Ablation Study: Even when inputs were transformed (semantically equivalent code), MetaCompress consistently identified behavioral fidelity gaps, proving the framework is robust to input variations.

5. Significance and Implications

• Beyond Accuracy: The paper fundamentally shifts the evaluation paradigm for model compression. It proves that "good accuracy" does not equal "good compression" if the student model lacks the teacher's robustness and decision consistency.
• Safety and Reliability: For safety-critical applications (e.g., vulnerability detection), deploying a student model with high MR violation rates is risky, as it may fail unpredictably under real-world code variations or adversarial conditions.
• Actionable Guidance:
  • Practitioners: Should use MetaCompress violation rates as a gating criterion for deployment. Low violation rates are acceptable for non-critical tasks; high rates require architectural changes or additional safeguards.
  • Researchers: Highlights the need for robustness-aware distillation. Future KD methods should move beyond aligning final logits to aligning intermediate representations, confidence distributions, and calibration behaviors to truly close the "knowledge gap."
• Framework Utility: MetaCompress provides a practical, low-overhead tool (leveraging existing forward passes) to diagnose behavioral drift in compressed models without requiring ground-truth labels for every test case.

In conclusion, the paper establishes that current knowledge distillation techniques for code models often result in "shallow mimicry." MetaCompress offers the necessary tools to detect and quantify this shallow mimicry, ensuring that compressed models are not just accurate, but also robust and behaviorally faithful to their teachers.