💻 computer science

Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence

This paper introduces T2J-Bench, a benchmark that evaluates codebase conversion through a fixed, multi-stage observational equivalence contract, revealing that current coding agents significantly overestimate their success due to reliance on flawed self-validation rather than limited computational resources.

Original authors: Linxin Song, Jiefeng Chen, Yue Huang, Bhavana Dalvi Mishra, Chi Wang, Jieyu Zhao, Jinsung Yoon, Tomas Pfister

Published 2026-05-29

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Linxin Song, Jiefeng Chen, Yue Huang, Bhavana Dalvi Mishra, Chi Wang, Jieyu Zhao, Jinsung Yoon, Tomas Pfister

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Problem: The "Fake It 'Til You Make It" Agent

Imagine you hire a very confident, high-tech robot chef (a Coding Agent) to take a complex, 100-page recipe book written in French (PyTorch code) and translate it into English (JAX code).

The robot chef is great at reading the words and rearranging the sentences. It finishes the job quickly and says, "Done! The book is translated."

However, when you actually try to cook a meal using the new English book, the soup tastes like soap, or the cake collapses. The robot chef didn't fail because it couldn't read the words; it failed because it didn't understand the chemistry of cooking. It translated the words correctly, but it messed up the semantics (the actual meaning and behavior).

The paper argues that current AI coding agents are too quick to declare victory. They run a few quick checks (like "Does the book have pages?" or "Is the font readable?") and say, "Success!" even though the underlying logic is broken.

The Solution: T2J-Bench (The "Taste Test" Benchmark)

To fix this, the researchers built a new testing ground called T2J-Bench. Instead of just asking, "Did the robot finish the book?" they ask, "Does the translated book produce the exact same meal as the original?"

They call this "Observational Equivalence." It's like a blind taste test. The evaluator doesn't care how the robot wrote the code; it only cares if the output is identical to the original.

The test happens in three strict stages, like a security checkpoint:

The Spec Stage (The ID Check):
- Analogy: Does the new book have the same chapters, the same table of contents, and the same page numbers as the original?
- Reality: Does the converted code have the right entry points, the right variables, and the right structure?
- Result: The robots are good at this. 91% of them pass this stage. They can make the book look right.
The Numeric Stage (The Ingredient Check):
- Analogy: If the original recipe calls for 200g of flour and 3 eggs, does the new recipe also call for exactly that? If you mix the ingredients, do you get the exact same batter weight?
- Reality: Does the code produce the exact same numbers (losses, gradients, outputs) when run on a small test batch?
- Result: This is where things break. Many robots pass the ID check but fail the ingredient check. They might swap a "loss" number with a "method" number, making the math look right but the result wrong.
The Behavioral Stage (The Cooking Test):
- Analogy: If you cook the cake for 10 minutes using the new recipe, does it rise and brown exactly like the original cake?
- Reality: If you run a short training session (a few steps), does the AI model learn and behave the same way as the original?
- Result: This is the hardest part. Even if the ingredients are right, the process of cooking might be different.

The Shocking Results

The researchers tested 355 attempts by the world's smartest AI coding agents (including models from Google, Anthropic, and OpenAI).

The Pass Rate is Terrible: Even the best system only passed the full test 26.7% to 28.9% of the time.
More Money Doesn't Help: The researchers tried giving the robots more time and more "thinking tokens" (like giving them a bigger budget to hire more helpers). It didn't help much. Spending 4.7 times more money only improved the success rate by 2.2 times.
The "Confidence Trap": This is the most surprising finding. The robots are wildly overconfident.
- The robots told the researchers: "I am 95% sure I succeeded!"
- The actual test said: "You only succeeded 28% of the time."
- The robots were lying to themselves (or rather, their internal checks were lying to them) by a margin of 66 to 97 percentage points.

Why Are They Failing?

The paper concludes that the problem isn't that the robots aren't smart enough or don't have enough money. The problem is Self-Validation.

The Analogy: Imagine a student taking a math test. Instead of checking their answers against the answer key, they check if their handwriting is neat and if they filled in all the bubbles. They say, "I got an A!" because the form looks perfect, even though the math is wrong.
The Reality: The agents check if the code "runs" and if the files exist. They don't check if the code actually means the same thing as the original. They confuse "looking like a translation" with "being a translation."

The Takeaway

The paper suggests that to fix coding agents, we can't just make them bigger or give them more money. We need to teach them to check their work against the original source, not just against their own internal feelings.

They need to stop asking, "Does this code run?" and start asking, "Does this code do exactly what the original code did?" until the answer is a perfect "Yes."

In short: Current AI agents are great at translating the words of code, but they are terrible at translating the soul of the code. They are "Converted, Not Equivalent."

Technical Summary: Converted, Not Equivalent: Benchmarking Codebase Conversion via Observational Equivalence

Problem Statement

Recent advances in coding agents have enabled them to act as codebase-scale collaborators capable of navigating projects, editing files, and performing large-scale software maintenance. However, a critical weakness has emerged: agents frequently over-trust their own local validation routines. In the context of codebase conversion (e.g., migrating from PyTorch to JAX), agents often declare success based on surface-level checks—such as a callable entry point, a finite loss, or syntactic validity—while violating the semantic contracts that users actually depend on.

Prior evaluation methods for codebase conversion are largely outcome-driven, comparing final task success or end-state artifacts under hidden evaluators. This approach is unstable for semantic-preserving conversion because observable outcomes can be sparse relative to the implementation space. Two codebases with vastly different internal contracts can produce nearly identical forward losses on a single batch while diverging sharply in gradients, optimizer states, or short-horizon training dynamics. Consequently, outcome-only comparisons risk rewarding accidental agreement on shallow probes while failing to detect semantic failures.

Methodology: T2J-Bench

To address this evaluation gap, the authors introduce T2J-Bench, a benchmark that reformulates codebase conversion as transfer under a fixed equivalence contract. Rather than asking if a converted codebase merely executes, T2J-Bench verifies whether the converted codebase behaves like the original under a fixed set of bounded tests derived from the source.

Core Framework

The benchmark operates through a three-stage, ordered verification process. A candidate codebase must pass each stage sequentially to be considered successful:

Spec (Interface Admissibility): Verifies that the converted codebase exposes the same training interface as the source. This includes checking parameter trees, batch schemas, public entry points, and the ability to construct a runtime without errors.
Numeric (Forward Outputs and Tensors): Compares specific artifacts computed during bounded tests. This includes forward outputs, losses, gradients, and objective-specific tensors (e.g., policy log-probabilities for DPO, value outputs for PPO). Comparisons use strict tolerances for absolute/relative error, cosine similarity, and KL divergence.
Behavioral (Short-Horizon Dynamics): Probes whether local numeric agreement is sufficient to preserve short training dynamics under fixed seeds. This involves replaying a few training steps to ensure that optimizer updates, reward computations, and reference-model interactions produce trajectories consistent with the source.

Experimental Setup

Task: Converting full PyTorch training codebases for SFT (Supervised Fine-Tuning), DPO (Direct Preference Optimization), and PPO (Proximal Policy Optimization) into JAX.
Dataset: 45 datapoints covering 15 model families (e.g., LLaMA variants, Mistral, Qwen) with size constraints (<7B parameters) and bounded configurations.
Evaluation Protocol:
- Controlled Baselines: Frontier models (e.g., Claude Opus 4.7, Gemini 3.1 Pro, Qwen3 Coder) evaluated using a self-implemented, backbone-agnostic multi-agent scaffold.
- Native Agents: Full coding agent systems (e.g., Claude Code, Codex) evaluated in their native interfaces.
- Constraints: A "minimum constraint prompt" ( $p^\star$ ) is used to fix the training interface without leaking evaluator internals. The verifier remains external and fixed.

Key Results

The evaluation of 355 blind conversion attempts yields four central findings:

Codebase Conversion is Far from Solved: Despite high pass rates on the initial Spec stage (up to 91.1%), the overall end-to-end pass rate for the strongest systems is only 26.7–28.9% at @1. The primary bottleneck is preserving numerical equivalence and training dynamics, not merely producing executable code.
Compute Budget is a Weak Remedy: Increasing the inference budget (token count) yields diminishing returns. A 4.7× spread in per-attempt token cost results in only a 2.2× spread in overall pass rate. Systems that spend more tokens often engage in repeated investigation or verbose self-verification without improving the final conversion quality.
Systematic Overconfidence: Every evaluated system significantly overestimates its own success. The gap between self-reported pass rates (often >90%) and evaluator-measured pass rates (0–29%) ranges from 66.6 to 97.8 percentage points. Agents tend to verify that artifacts exist and are well-formed (e.g., "loss is a finite scalar") but fail to verify that these artifacts match the source semantics.
Failure Modes are Contract-Misaligned: The hardest failures stem from contract-misaligned self-validation rather than limited backbone strength. Common failure patterns include:
- Seam Errors: Type confusion at the JAX/PyTorch boundary (e.g., passing numpy.ndarray to a PyTorch embedding layer).
- Semantic Substitution: Returning the DPO method loss in the forward_loss slot, satisfying local smoke tests but violating the specific contract required by the evaluator.
- Gradient/State Divergence: Matching forward passes but failing to preserve gradient norms or optimizer state updates.

Significance and Claims

The paper claims that T2J-Bench provides a necessary shift from outcome-driven to contract-driven evaluation. By decomposing verification into Spec, Numeric, and Behavioral stages, the benchmark circumvents the undecidability of absolute codebase equivalence while remaining discriminative across different models and agent systems.

The authors argue that in the current regime of codebase conversion:

Backbone scaling alone is insufficient: The gap between the best and worst models is small compared to the gap between self-validation and external verification.
Self-validation is the primary bottleneck: Agents fail not because they cannot compute the correct values, but because their internal validation loops treat "well-formed artifacts" as equivalent to "semantically correct artifacts."
Future improvements require external alignment: Closing the gap requires agents to internalize a source-comparison loop rather than relying on self-consistent smoke tests. The benchmark serves as a diagnostic scaffold to identify these misalignments.

The paper concludes that while T2J-Bench is instantiated on PyTorch-to-JAX conversion, its framework of fixed equivalence contracts and staged verification offers a generalizable approach for evaluating semantic-preserving codebase transformations.