💻 computer science

False Proximity: Frozen Code Embeddings Invert Similarity on Behaviourally Divergent Mutant Pairs

This paper demonstrates that frozen CodeBERT embeddings systematically invert similarity signals for behaviorally divergent mutant pairs, incorrectly scoring them as more similar than genuinely equivalent code, thereby revealing that such embeddings reflect surface proximity rather than behavioral equivalence and necessitate contrastive fine-tuning or execution-based verification for reliable code analysis.

Original authors: Keletso J. Letsholo, Nourchene Elleuch Ben Ayed, Achraf Ghorbel, Muath Alrammal

Published 2026-10-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Keletso J. Letsholo, Nourchene Elleuch Ben Ayed, Achraf Ghorbel, Muath Alrammal

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the vast landscape of modern software, programs are often treated as if they speak a language that computers can instantly understand and compare. To make sense of this complexity, researchers have developed tools that translate lines of code into mathematical points in a high-dimensional space. Imagine these points as locations on a map; if two pieces of code are similar in what they do, their points should sit close together. This method, known as embedding, has become a standard way to search for code, find duplicate programs, or predict errors without running the software. The most popular tools for this task rely on models trained on massive amounts of text and code, learning to recognize patterns in how words and symbols are arranged. The prevailing assumption has been that if two programs look very similar on the surface—sharing the same words and structure—they must be doing the same thing. This belief underpins many automated systems that help developers manage their code, but it rests on a fragile intuition: that visual similarity always equals functional similarity.

A team of researchers set out to test this intuition with a rigorous experiment, asking a simple but dangerous question: what happens when two programs look almost identical but do the exact opposite? They constructed a controlled environment using thousands of real-world programming problems. From a collection of 800 distinct challenges, they gathered pairs of code that were known to be correct and functionally equivalent, meaning they solved the problem in the same way. Then, they created a second set of pairs by taking a correct program and making a tiny, single change to it—flipping a condition from "yes" to "no," swapping a "greater than" sign for a "less than" sign, or reversing the order of a list. These changes were small enough that the code looked nearly the same, but they were severe enough to make the program fail every single test case it was run against. The researchers then fed both sets of pairs into the standard code-comparison tools to see if the tools could tell the difference between a correct program and its broken, inverted twin.

The results were not just a failure to distinguish; they were a complete reversal of reality. When the tools compared the broken, inverted programs to their original versions, they rated them as more similar than they rated two completely different, independently written correct solutions to each other. In fact, the tools were wrong in more than nine out of ten comparisons. The mathematical score that was supposed to measure how well two programs matched was actually measuring how much they looked alike on the surface. Because the broken programs shared almost every word and line with the original, the tools saw them as perfect matches, completely missing the fact that their behavior was diametrically opposed. This phenomenon occurred consistently across different types of code changes and different versions of the comparison tools, suggesting that the problem is not a minor glitch but a fundamental flaw in how these models interpret code when they are not specifically trained to look for behavioral differences.

The researchers then investigated whether this error could be fixed by simple, low-cost adjustments that do not require retraining the massive models. They tried mathematical tricks designed to spread the code points out more evenly in their space, hoping this would make the differences more visible. They also tried combining the code comparison with a structural analysis that counts how many lines of the code tree are different. None of these methods worked. In fact, adding the structural analysis made the problem worse. Because the broken programs were structurally almost identical to the originals—differing by only a single line—the structural tool gave them a high similarity score, reinforcing the error rather than correcting it. The tools were so focused on the surface appearance of the code that they could not see the functional inversion, even when the researchers explicitly tried to force them to look deeper.

This study reveals a critical blind spot in the technology that powers modern code search and duplicate detection. The tools are excellent at finding code that looks the same, but they are dangerously misleading when asked to judge whether two pieces of code do the same thing, especially when one is a subtle, broken version of the other. The researchers found that for these specific types of errors, the standard approach of using frozen, pre-trained models is not just imprecise; it is inverted. To reliably tell the difference between a correct function and its behavioral opposite, developers cannot rely on these similarity scores alone. Instead, they must either retrain the models with specific examples of these errors or, more reliably, actually run the code to see what it does. The study concludes that until these changes are made, systems relying on these similarity scores will continue to mistake a broken program for a perfect match, potentially hiding critical bugs in plain sight.

Technical Summary: False Proximity in Frozen Code Embeddings

Problem Statement
Code search, clone detection, and defect prediction pipelines frequently rely on cosine similarity over frozen embeddings from pre-trained models like CodeBERT as a lightweight proxy for behavioral equivalence. The prevailing intuition is that semantically close code fragments will yield high cosine similarity scores. However, this intuition has not been rigorously validated for "polarity-sensitive" scenarios where two code fragments compute opposite results (e.g., a function returning the maximum of two numbers versus one returning the minimum).

This paper investigates whether frozen code embeddings can distinguish between behaviorally equivalent pairs (independently written solutions to the same problem) and behaviorally divergent mutant pairs (single-site mutations that invert the program's output). The authors hypothesize that because mutant pairs are lexically and structurally near-identical to their originals, while equivalent pairs are stylistically diverse, embedding models trained on masked language modeling (without polarity-distinguishing signals) may fail to separate these classes. Specifically, they test whether the similarity signal merely fails (random noise) or inverts (ranking divergent mutants as more similar than equivalent pairs).

Methodology
The study constructs a controlled benchmark derived entirely from Project CodeNet Python800 to eliminate domain-shift confounds.

Dataset Construction:
- Source: 800 problems, each with 300 judge-accepted Python submissions.
- Equivalent Pairs: 800 pairs of independently written, execution-verified solutions (732 oracle-verified, 67 agreement-verified).
- Mutant Pairs: 1,833 pairs generated via single-site Abstract Syntax Tree (AST) transformations. Mutants are retained only if they are "killed" (output diverges from the original on every usable oracle input).
- Mutation Categories:
  - COD (Conditional Operator Deletion/Insertion): Inverting conditions (e.g., if x to if not x).
  - ROR (Relational Operator Replacement): Replacing operators with logical complements (e.g., < to >=).
  - AI (Algorithmic Inversion): Reversing primary operations (e.g., min/max swap, sort direction reversal).
- Analysis Set: 2,617 pairs across 784 problems where both equivalent and mutant pairs exist.
- Validation: Inter-rater reliability (IRR) by two external PhD annotators yielded Cohen's $\kappa = 0.925$ .
Experimental Design:
- Models: Three frozen encoders: CodeBERT, GraphCodeBERT, and UniXcoder.
- Configurations: Eight configurations total, including CodeBERT layers 6, 9, and 12 with both [CLS] and mean pooling.
- Metrics:
  - AUC-ROC: Probability that an equivalent pair scores higher than a mutant pair. Chance = 0.5; < 0.5 indicates systematic inversion.
  - False-High-Similarity Rate (FHSR): Proportion of mutant pairs scoring above the 75th percentile of the equivalent-pair distribution.
  - Jensen-Shannon Divergence (JSD): Measures separation magnitude between distributions.
- Training-Free Corrections: The study evaluates whether isotropy corrections (whitening, all-but-the-top) or a frozen contrastively trained encoder (CodeSage) can repair the failure.
- Hybrid Model (RQ3): A training-free hybrid combining cosine similarity with normalized AST edit distance (using APTED) to test if structural signals can correct the polarity error.

Key Results

Systematic Inversion (RQ1): Across all eight embedding configurations, the models fail to discriminate. The AUC is far below the 0.5 chance level, ranging from 0.017 (UniXcoder) to 0.090 (CodeBERT CLS, Layer 12). This indicates that in roughly 9 out of 10 comparisons, a behaviorally divergent mutant is scored as more similar to its original than two independently written correct solutions are to each other.
- The FHSR is consistently high (0.870 to 0.978), meaning nearly all mutant pairs exceed the threshold used to identify "similar enough" code.
- The failure persists even when controlling for token overlap (matched subsets still show AUC < 0.5) and within shared-anchor comparisons.
Uniformity Across Categories (RQ2): The inversion is uniform across mutation types. COD, ROR, and AI exhibit statistically indistinguishable confusion rates (FHSR $\approx$ 0.87 for all). While token overlap contributes to the confusion, it does not fully explain it; the embedding geometry itself drives the proximity.
Failure of Training-Free Corrections (RQ1 & RQ3):
- Isotropy Corrections: Applying whitening or "all-but-the-top" transforms to the embeddings worsened the discrimination (AUC dropped to near 0.004–0.024), as removing the anisotropic component exposed the surface-driven proximity.
- Contrastive Encoder: A frozen, contrastively trained encoder (CodeSage) also inverted the similarity (AUC = 0.016), suggesting that standard contrastive objectives without specific polarity-negative pairs are insufficient.
- Hybrid Model: Combining cosine similarity with AST edit distance (RQ3) deepened the failure. Because single-site mutants are structurally near-identical to their originals (AST distance $\approx$ 0), the structural term assigns them high similarity, reinforcing the incorrect ranking. The best hybrid model achieved an AUC of 0.002, and the FHSR reached 1.000.

Significance and Claims
The paper claims that the failure is not an inherent flaw in CodeBERT's ability to perform its original tasks (code search, clone detection of Type-1/2 clones), but rather a consequence of applying frozen embedding similarity to polarity-sensitive equivalence decisions.

Surface Proximity vs. Behavioral Equivalence: High cosine similarity in this context reflects surface proximity (lexical and structural near-identity) rather than behavioral agreement.
Limitations of Heuristics: The study demonstrates that training-free fixes—whether geometric (isotropy corrections) or structural (AST distance)—cannot repair this specific inversion. In fact, structural similarity acts as a confounding factor that reinforces the error because mutants are structurally near-twins of their originals.
Practical Implication: Practitioners relying on frozen CodeBERT embeddings for code search, clone screening, or defect prediction involving near-duplicate code with inverted logic should not trust high similarity scores as evidence of functional equivalence. The paper concludes that contrastive fine-tuning (with specific polarity-negative pairs) or execution-based verification are the necessary routes to resolve this ambiguity, as no training-free signal currently available can distinguish these pairs.

The findings are scoped to Python and the specific frozen encoder models tested. The authors note that while the inversion is established for this controlled benchmark, the applicability to production code or other languages remains an open empirical question.

Technical Summary: False Proximity in Frozen Code Embeddings

More like this