Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agents

This paper presents a comprehensive survey of 178 benchmarks for Code Large Language Models and Agents through a tiered Software Development Life Cycle (SDLC) framework, revealing a significant imbalance that heavily favors the implementation phase while neglecting requirements and design, alongside critical gaps in anti-contamination strategies that necessitate future research to bridge the gap between theoretical capabilities and practical effectiveness.

The Gist

Imagine the world of software development as a massive, complex construction project. You have the Software Development Life Cycle (SDLC), which is like the blueprint for building a skyscraper. It has distinct stages:

1. Requirements: Figuring out what the building needs to do.
2. Design: Drawing the blueprints.
3. Implementation: Actually laying the bricks and pouring the concrete (writing the code).
4. Testing: Checking for cracks and making sure the lights work.
5. Maintenance: Fixing leaks and renovating years later.

Now, enter the Code Large Language Models (CodeLLMs). Think of these as super-intelligent, AI-powered construction workers. They can read your instructions and build things incredibly fast.

This paper is essentially a report card audit for the "tests" we use to see how good these AI workers are. The authors looked at 178 different tests (benchmarks) used by researchers and found some major problems with how we are grading these AI workers.

Here is the breakdown in simple terms:

1. The "Gym Class" Imbalance

The biggest finding is that our tests are completely unbalanced.

• The Problem: Imagine if you only tested a construction worker on how fast they can lay bricks (Implementation), but never asked them to read the blueprints (Design), figure out what the client wants (Requirements), or fix a broken pipe later (Maintenance).
• The Reality: The paper found that 61% of all tests focus only on "laying bricks" (writing code).
• The Neglect: Only 5% of tests check if the AI can understand what the client actually wants, and a tiny 3% check if they can design the building.
• The Metaphor: It's like a driving school that only tests if you can parallel park, but never tests if you can read a map, follow traffic signs, or handle a flat tire.

2. The "Cheating" Problem (Data Contamination)

• The Problem: To get good at a test, you have to study for it. But in the AI world, the "study material" is the internet.
• The Metaphor: Imagine a student taking a math test. If the teacher accidentally left the answer key on the desk, and the student memorized it before the test, they would get a perfect score. But they didn't actually learn math; they just memorized the answers.
• The Reality: Many of these AI tests use old code that the AI models have already seen while they were being trained. The AI isn't "thinking"; it's just recalling what it saw before. This makes the AI look smarter than it really is. The paper notes that very few tests have good "anti-cheating" strategies to stop this.

3. The "Robot vs. Human" Gap

• The Problem: Most tests are set up like a simple game of "Question and Answer." You ask a question, the AI gives an answer, and you grade it.
• The Metaphor: Real construction isn't a quiz. It's a conversation. You say, "Build a wall," the worker builds it, you say, "No, make it higher," and they fix it. They might need to grab a hammer, check a level, or talk to the electrician.
• The Reality: Current tests mostly check if the AI can give a single correct answer. They rarely test if the AI can work in a team, use tools, or fix mistakes over time (which is what "Agents" are supposed to do).

4. The Language Bias

• The Problem: The tests are heavily biased toward Python (a popular programming language).
• The Metaphor: It's like a driving test that only happens on sunny days with smooth asphalt. But what if the AI has to drive in the rain, on gravel, or in a different country?
• The Reality: We don't know if these AI workers are good at building with "Rust" or "Go" (other modern languages) because we haven't tested them enough in those environments.

What Should We Do Next?

The authors suggest we need to upgrade our "driving tests" for AI:

• Test the whole cycle: Don't just test coding; test planning, designing, and fixing.
• Stop the cheating: Create fresh, new tests that the AI hasn't seen before.
• Test the real world: Instead of simple quizzes, give the AI complex projects where they have to use tools, talk to other systems, and fix their own mistakes over time.
• Include privacy: Make sure the AI doesn't accidentally leak secret company data while working.

In a nutshell: We have built some amazing AI construction workers, but we are currently only testing them on how fast they can stack blocks. We need to start testing them on how well they can design, build, and maintain entire skyscrapers in the real world.

Technical Summary

Here is a detailed technical summary of the paper "Software Development Life Cycle Perspective: A Survey of Benchmarks for Code Large Language Models and Agents."

1. Problem Statement

Code Large Language Models (CodeLLMs) and AI agents are rapidly transforming software engineering, moving beyond simple code generation to support complex tasks across the entire Software Development Life Cycle (SDLC). However, the current landscape of benchmarks used to evaluate these models suffers from three critical issues:

• Structural Imbalance: Existing benchmarks are heavily skewed toward the Software Implementation phase (coding), while critical early stages like Requirements Engineering and Software Design are severely underrepresented.
• Lack of Realism and Depth: Many benchmarks rely on synthetic data, single-turn interactions, and static evaluation metrics that fail to capture the iterative, multi-step, and context-rich nature of real-world engineering workflows.
• Data Contamination Risks: A significant portion of benchmarks lacks effective anti-contamination strategies, leading to data leakage where test data is inadvertently included in model training corpora, resulting in inflated performance assessments.

There is a lack of comprehensive reviews that analyze benchmarks specifically through the lens of the SDLC to identify these gaps and guide future research.

2. Methodology

The authors conducted a systematic literature review and analysis using a rigorous four-stage protocol:

• Data Collection: They searched four major academic databases (IEEE Xplore, ACM Digital Library, Elsevier, Springer) and Google Scholar using 20 keyword combinations related to CodeLLMs, agents, and specific SDLC phases.
• Filtering & Snowballing: From an initial pool of 1,347 papers, they filtered for relevance (LLM applications in SE, published since 2022, influential venues) to select 321 papers. They then performed forward and backward snowball sampling, adding 140 more papers, resulting in a final set of 461 papers.
• Benchmark Extraction: From these papers, they identified and analyzed 178 distinct benchmarks.
• Tiered Analysis Framework: The authors proposed a novel framework to categorize and evaluate these benchmarks:
  • General Dimensions: Applicable across all phases (Anti-Contamination Strategy, Authenticity of data, Interaction Mode, Evaluation Paradigm).
  • Stage-Specific Dimensions: Tailored to specific SDLC phases (e.g., Requirement Evolution for Requirements Engineering; Context Scope for Implementation; Evolution for Maintenance).
• Mapping: Benchmarks were mapped to five SDLC phases: Requirements Engineering, Software Design, Software Implementation, Software Testing, and Software Maintenance.

3. Key Contributions

• Comprehensive Taxonomy: The paper provides the first systematic categorization of 178 CodeLLM benchmarks across the full SDLC, revealing a stark imbalance in coverage.
• Tiered Analysis Framework: Introduction of a dual-layer framework (General + Stage-Specific dimensions) to assess the alignment of benchmarks with real-world engineering needs.
• Identification of Critical Gaps: Detailed analysis of deficiencies in current benchmarks, particularly regarding data leakage, lack of multimodal support, and insufficient evaluation of iterative/agentic workflows.
• Open-Source Resource: The authors curated a standardized knowledge base with detailed metadata for all 178 benchmarks, made available in a public repository.

4. Key Results and Findings

The analysis yielded several quantitative and qualitative insights:

A. Structural Imbalance in SDLC Coverage

• Software Implementation: Dominates the field, accounting for 61% of all benchmarks (109 out of 178). This includes code generation, completion, and text-to-SQL.
• Software Testing: Accounts for 12% (22 benchmarks).
• Software Maintenance: Accounts for 15% (27 benchmarks).
• Requirements Engineering: Only 5% (9 benchmarks), with a notable absence of benchmarks for Requirements Management.
• Software Design: Only 3% (5 benchmarks), with zero public benchmarks for Architectural or Database Design.

B. Critical Deficiencies

• Anti-Contamination: Most benchmarks lack robust anti-contamination strategies. Static defenses (e.g., time-cutoffs) are failing as training corpora expand. Dynamic, continuous updates are rare.
• Interaction Modes: The majority of benchmarks rely on single-turn interactions. There is a severe lack of benchmarks supporting Agentic workflows (multi-turn, tool use, environment interaction), which are essential for modern agents.
• Authenticity & Context:
  • Implementation: While moving toward repository-level tasks, many still rely on synthetic data or lack cross-file context.
  • Design/Requirements: Benchmarks often use plain text, ignoring multimodal inputs (diagrams, UI mockups) and failing to evaluate requirement evolution.
  • Maintenance: Benchmarks often focus on single-file repairs, ignoring the evolutionary nature of maintaining complex, legacy systems.
• Language Bias: Python is the dominant language (103 benchmarks), followed by Java and C/C++. Modern languages like Rust and Go are significantly underrepresented, and multilingual benchmarks often rely on translation rather than native corpora.

C. Specific Phase Observations

• Requirements: Existing benchmarks fail to evaluate the ability to handle conflicting or evolving requirements.
• Design: Current benchmarks focus on retrieval or summarization rather than generative design creativity.
• Testing: Heavy reliance on C/C++ and Java; limited support for modern languages and dynamic execution environments.
• Maintenance: Lack of benchmarks for "evolution-aware" repair (adapting to changing requirements over time).

5. Significance and Future Directions

This survey serves as a foundational resource for the research community, shifting the focus from evaluating CodeLLMs as mere "coders" to evaluating them as "Software Engineers."

Proposed Future Directions:

1. Standardize Early-Phase Benchmarks: Develop rigorous benchmarks for Requirements Engineering and Design that support multimodal inputs and iterative refinement.
2. Agent-Centric Evaluation: Transition from static, single-turn QA to agentic evaluation paradigms that test tool use, environment interaction, and multi-step reasoning.
3. Dynamic Anti-Contamination: Move away from static time-cutoffs toward continuous, online benchmark updates to prevent data leakage.
4. Real-World Maintenance: Create benchmarks that simulate long-term software evolution, including dependency management and configuration breakages.
5. Human-AI Collaboration & Privacy: Introduce metrics for human-in-the-loop efficiency and privacy preservation, which are critical for enterprise adoption.
6. Language Diversity: Build native, high-quality corpora for modern languages (Rust, Go) rather than relying on translated datasets.

In conclusion, the paper argues that for CodeLLMs and agents to be truly effective in industry, the benchmarking ecosystem must evolve to match the complexity, diversity, and iterative nature of the full Software Development Life Cycle.