💬 NLP

SlopCodeBench: Benchmarking How Coding Agents Degrade Over Long-Horizon Iterative Tasks

The paper introduces SlopCodeBench, a benchmark demonstrating that current coding agents systematically fail at long-horizon iterative tasks by exhibiting steady degradation in code quality through increased verbosity and structural erosion, a problem that pass-rate benchmarks fail to capture.

Original authors: Gabriel Orlanski, Devjeet Roy, Alexander Yun, Changho Shin, Alex Gu, Albert Ge, Dyah Adila, Frederic Sala, Aws Albarghouthi

Published 2026-03-27

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Gabriel Orlanski, Devjeet Roy, Alexander Yun, Changho Shin, Alex Gu, Albert Ge, Dyah Adila, Frederic Sala, Aws Albarghouthi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you hire a very talented, fast-talking architect to build a house.

The Old Way (Current Benchmarks):
You give the architect a blueprint for a small cottage. They build it perfectly. You walk through, check the doors and windows, and say, "Great job! 100% pass rate!" You hire them again for a bigger house, and they do it again.
The problem: The benchmarks only check if the house stands up right now. They don't ask: "If we need to add a second floor next year, will the foundation collapse? Will the walls be so messy with duct tape and glue that we can't drill a hole without the whole thing falling apart?"

The New Way (SlopCodeBench):
This paper introduces a new test called SlopCodeBench. Instead of building a house from scratch every time, the architect has to keep adding rooms to the same house they built yesterday, based on new, changing requests.

The Request: "Okay, now add a kitchen." (They build it).
The Twist: "Now, make the kitchen support a heavy stone oven, and also add a secret door."
The Twist 2: "Now, we need to add a library, but the stone oven is blocking the way, so you have to move it."

The "Slop" Problem

The researchers found that as these AI coding agents keep working on the same project, the code starts to turn into "Slop."

Think of "Slop" like a messy kitchen counter where you keep piling up ingredients without cleaning up.

Verbosity (The Clutter): The agents start writing way too much code. Instead of saying "Add salt," they write a 50-page essay on why salt is important, how to find it, and three different ways to pour it. It works, but it's bloated and hard to read.
Structural Erosion (The Rot): This is the scary part. Imagine the house's main support beam. At first, it's a clean, strong steel beam. As the agent adds more rooms, they start nailing new things directly onto that one beam. They don't build new supports; they just pile more weight onto the old one. Eventually, that one beam is holding up the entire roof, the kitchen, and the library. It's a "God Function"—a single piece of code so huge and tangled that if you touch it to fix a small bug, the whole house might shake.

What They Discovered

The team tested 11 different AI models (like the smartest coding assistants available) on 20 different software projects.

No One Won: Not a single AI could finish a project from start to finish without the code getting messy. The best one only solved about 17% of the final steps correctly.
The "Slop" Grows: As the projects got longer, the code got messier.
- 80% of the time, the code became "eroded" (the messy, overloaded beams got worse).
- 90% of the time, the code became "verbose" (too much clutter).
AI vs. Humans: The researchers compared the AI's messy code to real, maintained software written by humans (like the code behind popular apps).
- Human code stays relatively flat and clean over time.
- AI code gets worse with every single update. It's like a human who cleans their room every week, while the AI is a teenager who just keeps throwing more clothes on the floor without ever picking anything up.

The "Magic Prompt" Experiment

The researchers tried to fix this by giving the AI a strict set of rules in the beginning, like: "Be concise! Don't make one giant function! Plan before you write!"

The Result: It helped a little at the very start. The first room was cleaner.
The Reality: But as soon as the AI started adding the second and third rooms, it forgot the rules. The mess started piling up again at the exact same speed. The "magic prompt" couldn't stop the degradation; it just delayed the inevitable.

The Big Takeaway

Current tests for AI coders are like a driving test where you only drive in a straight line for 10 seconds. The AI passes because it doesn't hit anything.

But real software development is like driving across the country for a week. You have to navigate traffic, bad weather, and detours. This paper shows that while AI is great at driving in a straight line, it gets lost, crashes, and leaves a trail of wreckage when it has to keep driving and adapting for a long time.

In short: AI can write code that works today, but it's terrible at writing code that can be extended tomorrow. The code becomes a "sloppy" mess that is hard to fix, hard to read, and eventually, impossible to maintain.

1. Problem Statement

Current coding agent benchmarks (e.g., SWE-bench) predominantly evaluate single-shot solutions against complete specifications. While these benchmarks measure if an agent can produce code that passes a test suite, they fail to capture a critical failure mode in real-world software engineering: iterative degradation.

In practice, software development is iterative. Agents must extend their own prior code under evolving specifications. Existing iterative benchmarks often fail to measure this because they either:

Decompose tasks into dependency-ordered subproblems where the agent doesn't own the architecture.
Use historical commit data where the agent inherits human-designed structures rather than making its own architectural decisions.
Rely on pass/fail metrics that remain stable even as code quality (maintainability, extensibility) collapses.

The paper argues that current agents generate "slop"—code that is verbose, duplicated, and structurally eroded—making it progressively harder to extend, even if it passes immediate tests.

2. Methodology: SlopCodeBench (SCBench)

The authors introduce SlopCodeBench, a language-agnostic benchmark designed to force agents to make architectural decisions and suffer the consequences of those decisions over time.

Benchmark Design

Structure: 20 distinct problems spanning 93 checkpoints.
Workflow: An agent starts from scratch at Checkpoint 1 (C1). At each subsequent checkpoint, the specification evolves (adding features or changing requirements), and the agent must modify its own prior workspace.
Constraints:
- No Prescribed Interfaces: Agents are given only external behavior (CLI/API contracts), not internal function signatures. They must design their own architecture.
- Black-Box Testing: The test suite is hidden. Agents see only specifications and examples, preventing them from "gaming" the tests or inferring internal structure from test code.
- State Persistence: The agent's code carries forward. A bad architectural choice at C1 becomes the foundation for C2, C3, etc.
Example Problem: code_search starts with Python-only regex matching (C1). Subsequent checkpoints add JavaScript/C++ support (C2), AST-based pattern matching (C3), and auto-fixes (C4). An agent that hardcodes logic at C1 faces cascading rewrites later.

Quality Metrics

Beyond correctness (pass rates), the authors introduce two trajectory-level metrics to quantify "slop":

Verbosity: Measures redundant or duplicated code growth.
- Calculated as the union of lines flagged by 137 targeted AST-Grep rules (detecting anti-patterns) and normalized clone lines, divided by total Lines of Code (LOC).
Structural Erosion: Measures the concentration of complexity in high-complexity functions.
- Defined as the fraction of the codebase's total "complexity mass" residing in functions with Cyclomatic Complexity (CC) > 10.
- Complexity Mass = $CC(f) \times \sqrt{SLOC(f)}$ .
- High erosion indicates that new logic is being patched into existing "god functions" rather than distributed into new, focused modules.

3. Key Contributions

SlopCodeBench: A novel benchmark comprising 20 problems and 93 checkpoints that isolates true iterative coding where agents own their architectural debt.
Quality Metrics: The introduction of Verbosity and Structural Erosion to track the compounding nature of code degradation, which standard pass-rate metrics miss.
Human Calibration: A comparative study against 48 maintained human Python repositories (ranging from niche to major projects like Django and scikit-learn).
Prompt Intervention Study: An analysis of whether "quality-aware" prompts (e.g., "Anti-Slop" or "Plan-First") can mitigate degradation.

4. Key Results

Performance and Correctness

Zero End-to-End Success: No agent solved any of the 20 problems end-to-end across all 11 evaluated models.
Low Solve Rates: The highest strict solve rate (passing all tests including regression) was 17.2% (Opus 4.6).
Regression Collapse: As checkpoints progress, the gap between "Core" pass rates (new features) and "Strict" pass rates (including old features) widens drastically. By the final checkpoint, strict solve rates collapse to near 0.5%.
Cost Inefficiency: Cost per checkpoint increased 2.9x from start to finish, but this spending did not improve correctness.

Quality Degradation ("Slop")

Universal Deterioration:
- Structural Erosion increased in 80% of trajectories.
- Verbosity increased in 89.8% of trajectories.
Compounding Effects: Agents tend to patch new logic into existing functions. For example, in one trajectory, a main() function grew from 84 lines (CC=29) to 1,099 lines (CC=285) with nine command branches repeating the same parsing logic.
Comparison to Humans:
- Agent code is 2.2x more verbose and significantly more eroded than maintained human repositories.
- While human code quality remains relatively flat or degrades slowly over time, agent code deteriorates with every iteration.
- Only 67% of human repos showed increased verbosity over time, compared to 43% growth in agents (median growth).

Prompt Interventions

Initial Improvement: Prompts instructing agents to "avoid slop" or "plan first" successfully reduced initial verbosity and erosion (lowering the intercept).
No Long-Term Fix: The slope of degradation remained unchanged. Once agents began iterating on their own code, the quality collapsed at the same rate regardless of the initial prompt.
No Correctness Gain: Despite cleaner code, pass rates did not improve, and costs often increased.

5. Significance and Conclusion

Limitations of Current Benchmarks: The paper demonstrates that pass-rate-centric benchmarks systematically undermeasure extension robustness. An agent can pass a test suite while producing code that is effectively unmaintainable for future iterations.
Architectural Discipline Gap: Current LLM agents lack the design discipline required for iterative software development. They favor local optimizations (getting the current test to pass) over global structural integrity, leading to rapid "structural erosion."
Future Directions: The results suggest that simply prompting for better code is insufficient. Future solutions likely require training-time interventions or tooling that enforces structural discipline across checkpoints, rather than just optimizing for immediate task completion.

In summary, SlopCodeBench reveals that while coding agents are capable of generating correct code for static tasks, they currently fail to sustain code quality in dynamic, long-horizon environments, accumulating technical debt ("slop") that renders their solutions increasingly brittle and expensive to maintain.