💻 computer science

ScarfBench: A Benchmark for Cross-Framework Application Migration in Enterprise Java

This paper introduces ScarfBench, a comprehensive benchmark for evaluating the ability of AI coding agents to perform behavior-preserving cross-framework migrations of enterprise Java applications across Spring, Jakarta EE, and Quarkus, revealing that current state-of-the-art models struggle significantly with this complex refactoring task.

Original authors: Advait Pavuluri, Bridget McGinn, Ashita Saxena, George Safta, Srikanth Tamilselvam, Raju Pavuluri, Michele Merler, Baishakhi Ray, Rahul Krishna

Published 2026-05-12

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Advait Pavuluri, Bridget McGinn, Ashita Saxena, George Safta, Srikanth Tamilselvam, Raju Pavuluri, Michele Merler, Baishakhi Ray, Rahul Krishna

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very old, well-built house. It's been lived in for decades, and it works perfectly for the family inside. But the neighborhood is changing. The old roads (the old software framework) are being paved over, and new, faster highways (modern frameworks) are being built. The family wants to move their furniture and keep their daily routine exactly the same, but they have to move into a house built on a completely different foundation with different plumbing, electrical wiring, and door handles.

This is the problem SCARFBENCH is trying to solve.

The Problem: Moving a House, Not Just Painting It

In the world of computer software, many big companies run their systems on "Java," a popular programming language. These systems are often built on three specific "frameworks" (think of these as different architectural styles or blueprints): Spring, Jakarta EE, and Quarkus.

Over time, companies want to switch from one style to another to save money or run faster. But you can't just swap the paint. You have to move the entire house.

The Challenge: It's not just changing a few words. You have to rewrite how the house is wired, how the doors open, and how the water flows, all while making sure the family inside doesn't notice a single difference in their daily life.
The Current State: Experts say this is incredibly hard. Until now, no one had a good way to test if Artificial Intelligence (AI) could do this job.

The Solution: SCARFBENCH (The "Moving Test")

The authors created a giant test called SCARFBENCH. Think of it as a "Moving Simulator" for software.

The Setup: They took 34 different "houses" (software applications) that were built in one style (e.g., Spring).
The Goal: They asked AI agents (smart computer programs) to rebuild these exact same houses in a different style (e.g., Quarkus or Jakarta), keeping the family's daily routine (the software's behavior) exactly the same.
The Proof: They didn't just ask the AI, "Did you finish?" They built a strict "Oral Exam" (a test suite). The new house had to:
- Pass the building inspector (compile).
- Turn on the lights and water (deploy).
- Let the family walk through the front door, sit in the living room, and turn on the TV exactly as they did before (pass behavioral tests).

The Results: The AI is Still Learning to Move

The researchers tested five of the smartest AI coding agents available today. Here is what they found:

The Score is Low: Even the best AI only managed to successfully move about 12% to 15% of the houses perfectly.
The "Fake Success" Trap: Many AIs could build a house that looked good on paper and even turned on the lights (it compiled and deployed), but when the family tried to use it, the doors didn't open or the water didn't flow. The AI could translate the words of the blueprint, but it couldn't understand the logic of the house.
Some Moves are Harder: Moving from one style to another was easier than others. Moving to the Jakarta style was the hardest; the AI failed to even get the house built (compile) in more than half the attempts.
One Success: Out of 204 different moving tasks, only one was completed perfectly by an AI, where the new house worked exactly like the old one.

Why This Matters

This paper doesn't say AI is useless. It says that while AI is getting good at writing code, it is still terrible at re-architecting complex systems. It's like having a robot that can write a poem, but if you ask it to move your entire kitchen to a new house and keep the fridge working, it might put the fridge in the bedroom.

The authors released this "Moving Simulator" (the benchmark) so that other researchers can use it to teach AI how to do these complex moves better in the future. They are essentially saying, "Here is a measuring stick; use it to see if your new AI can actually move the house, not just paint the walls."

In short: SCARFBENCH is a reality check. It shows that while AI is great at small tasks, moving entire software systems from one modern framework to another without breaking them is still a job that requires human-level understanding, and current AI is not quite there yet.

Technical Summary: SCARFBENCH

Problem Statement

Enterprise Java remains a dominant force in software infrastructure, with many applications outliving their original architectural assumptions. While modernization is often driven by deprecation pressures and non-functional goals like reduced memory overhead and improved cloud elasticity, migrating these applications across frameworks (specifically between Spring, Jakarta EE, and Quarkus) is a complex, risk-prone endeavor.

Current software engineering benchmarks primarily evaluate bug fixing, feature implementation, or language version upgrades within a fixed framework stack. They largely fail to measure cross-framework application refactoring, a task that requires more than local API substitution. Successful migration demands coordinated changes across build configurations, dependency injection, persistence, request handling, and deployment to preserve application behavior. Existing benchmarks often assume that preserving the underlying programming model is sufficient, an assumption that does not hold when moving between frameworks with divergent runtime models and idioms.

Methodology: SCARFBENCH

The authors introduce SCARFBENCH (Self-Contained Application Refactoring Benchmark), a benchmark designed to evaluate the ability of Large Language Model (LLM) agents to perform behavior-preserving cross-framework migrations.

Benchmark Construction

Scope: The benchmark comprises 34 application families implemented across three frameworks: Spring Boot, Jakarta EE, and Quarkus.
Structure: It is divided into two tiers:
1. Focused Tier: 29 self-contained applications isolating specific architectural layers (Dependency Injection, Persistence, Presentation, Infrastructure, Business Domain) anchored in Java Specification Requests (JSRs).
2. Whole-Application Tier: 5 larger, realistic open-source systems that combine these layers, testing cross-layer coupling and end-to-end workflows.
Variants: These applications yield 102 framework-specific variants (approximately 151K lines of paired Java code across 1,946 source and test files).
Tasks: The benchmark defines 204 directed migration tasks (e.g., Spring $\to$ Jakarta, Jakarta $\to$ Quarkus). Each task requires an agent to synthesize a target implementation that preserves the source behavior.
Ground Truth: Expert Java developers manually implemented all three framework variants for each application family, ensuring behaviorally equivalent implementations exist for validation.

Evaluation Protocol

Agents: Five state-of-the-art coding agents were evaluated: Claude Code (Opus 4.6), Gemini CLI (Gemini-3.1 Pro), Codex (GPT-5.4), Opencode (GLM-5.1), and Qwen CLI (Qwen3.5-397B-A17B).
Execution Harness: A containerized harness evaluates candidates in three sequential stages:
1. Compile: The candidate must build successfully.
2. Deploy: The application must start within a containerized runtime and emit readiness signals.
3. Behavioral Tests: The candidate must pass a suite of 1,331 expert-written behavioral tests.
Oracle: Correctness is determined by a framework-neutral behavioral oracle. Instead of unit tests (which would require migration themselves), the oracle uses Gherkin specifications concretized into Playwright/pytest scripts. These tests verify externally observable equivalence (HTTP routes, response payloads, UI flows, and persistent state changes).
Metrics: Success is measured by pass@1 at temperature 0, requiring the candidate to compile, deploy, and pass 100% of behavioral tests.

Key Contributions

SCARFBENCH Dataset: A curated corpus of 34 application families yielding 102 variants and 204 directed migration tasks, verified by 1,331 expert-written behavioral tests in a containerized environment.
Empirical Evaluation: A comprehensive assessment of five frontier coding agents, revealing significant gaps in current capabilities for system-level transformations.
Failure Taxonomy: A classification of 13 recurring failure categories spanning build, deploy, and test stages, derived from expert adjudication of failed traces.

Experimental Results

The evaluation reveals that behavior-preserving framework migration remains a significant challenge for current agents:

Low Success Rates: The strongest agent (Claude Code with Opus 4.6) achieved only 15.3% aggregate test pass on focused-layer migrations and 12.2% on whole applications. Across all 204 directed tasks, only one agent-produced migration passed its full test suite.
Build/Deploy vs. Behavior: There is a stark disconnect between surface-level success and behavioral correctness. Agents frequently produce code that compiles and deploys but fails behavioral tests. For instance, in whole-application migrations, compile success can reach 87% while behavioral success drops to 12%.
Asymmetric Difficulty: Migration difficulty is highly dependent on the target framework:
- Spring $\leftrightarrow$ Quarkus: The most tractable pair.
- Jakarta EE Targets: The most difficult. Only 2% of migrations targeting Jakarta EE pass behavioral tests, with 57% failing at the compile gate.
Prompt Engineering: "Skills-based" prompting (modularizing guidance) improved compile and deploy rates, particularly for focused tasks, but did not significantly close the gap in behavioral correctness for whole applications.

Failure Analysis

Analysis of failed tasks identified that errors extend beyond source-level translation. Common failure modes include:

Build Stage: Dependency resolution issues, stale module references, and project structure mismatches.
Deploy Stage: Resource/artifact mismatches (e.g., WAR vs. JAR packaging), incorrect runtime configuration (e.g., port mismatches), and broken dependency wiring.
Test Stage: Missing endpoints, incorrect response content, and infrastructure connectivity issues.
The authors note that Jakarta-targeted migrations fail disproportionately at the build gate, suggesting a divergence between learned framework idioms and Jakarta's packaging/server-configuration conventions.

Significance and Claims

The paper claims that while current agents can reliably translate framework APIs at the source level, they struggle to re-express application behavior through the target framework's runtime model. The results suggest that progress on repository-level benchmarks (like bug fixing or version upgrades) does not cleanly transfer to system-level transformations that require executing in a target runtime.

The authors position SCARFBENCH as a necessary step toward evaluating the "agentic" capabilities of LLMs in realistic enterprise scenarios, highlighting that true modernization requires more than syntactic substitution; it demands a deep understanding of architectural coupling, configuration semantics, and behavioral preservation. The benchmark, harness, and agent traces are released to facilitate further research into automated software modernization.