ClassEval-Pro: A Cross-Domain Benchmark for Class-Level Code Generation

1. Problem Statement

While Large Language Models (LLMs) have achieved high success in function-level code synthesis (e.g., HumanEval, MBPP) and repository-level code modification (e.g., SWE-bench), a critical gap remains in compositional code creation. This refers to the ability to build a complete, internally structured class from a specification, which requires coordinating multiple methods, managing shared state, and integrating logic across domains.

Existing benchmarks fail to address this effectively because:

• Function-level benchmarks test isolated logic synthesis.
• Repository-level benchmarks focus on modifying existing codebases.
• Prior class-level benchmarks (e.g., ClassEval) rely on manual curation, making them difficult to scale, expensive, and increasingly susceptible to data contamination (training on test data) due to their age (2023 vintage).

2. Methodology: ClassEval-Pro Construction

The authors introduce ClassEval-Pro, a benchmark comprising 300 class-level tasks across 11 domains. The construction relies on a fully automated, contamination-resistant three-stage pipeline:

A. Data Collection & Structuring (Stage 1)

• Source: Python repositories from GitHub updated after January 2025 to ensure the data is post-dated relative to current LLM training cutoffs.
• Domains: Covers 11 domains (e.g., Finance, Network, Security, Game Dev).
• Composition Strategy:
  • Intra-Domain: Merging classes within the same domain (e.g., SinglyLinkedList + Stack).
  • Cross-Domain: Merging classes from different domains (e.g., SQLQueryBuilder + PlayerInventory) to force complex state management and interface constraints.
• Filtering: Strict AST parsing ensures code is self-contained (standard library only), has 5+ methods, and 40–800 lines of code.

B. Test Generation & Filtering (Stage 2)

• Skeleton Generation: An LLM generates a class skeleton based on the merged code.
• Test Suite Generation: An LLM generates test cases matching the skeleton.
• Validation: An LLM Judge Ensemble (three independent judges) evaluates the compatibility between the skeleton and the test suite. Only pairs receiving full marks from at least two judges proceed.

C. Solution Generation & Verification (Stage 3)

• Reference Solution: A separate LLM generates the full solution code.
• Execution Verification: Solutions are run in an isolated environment. To be included in the benchmark, a task must:
  1. Pass all tests.
  2. Achieve >90% line coverage.
  3. Complete within a 60-second timeout.
• Final Output: A dataset of class skeletons, test suites, and verified reference solutions.

3. Experimental Design

• Models Evaluated: Five frontier LLMs (GPT-5.1, Gemini-2.5-Pro, Qwen3-480B, Qwen3-30B, Kimi-K2).
• Generation Strategies: Five distinct approaches were tested:
  1. Holistic: Single-pass generation of the entire class.
  2. Incremental: Iterative generation of methods in declaration order.
  3. Compositional: Independent generation of imports, class defs, and method bodies, then assembly.
  4. Top-Down: Generating high-level methods first, then dependencies.
  5. Bottom-Up: Generating low-level dependencies first, then high-level methods.
• Metrics: Primary metric is Pass@k (Class-level Full Success), with secondary analysis on Method-level success and Partial Success.

4. Key Results

A. Benchmark Difficulty & Discriminative Power

• Difficulty: ClassEval-Pro tasks are significantly harder than previous benchmarks, with deeper dependency hierarchies (peaking at 3 layers vs. 1-2 in ClassEval) and higher semantic diversity.
• Performance Gap: The best model (Qwen3-480B) achieved only 45.6% Pass@1 (Class-level Full Success). The gap between the strongest and weakest models was 17.7 percentage points, confirming the benchmark's ability to discriminate between frontier models.

B. Strategy-Model Interaction

• Structured Approaches Help Weaker Models: For GPT-5.1, the Bottom-Up strategy improved Pass@1 by 9.4 points (from 27.9% to 37.3%) compared to Holistic.
• Compositional Generation Fails: The "Compositional" strategy (generating parts in isolation) caused performance to collapse, dropping to as low as 1.3% for Gemini-2.5-Pro. This highlights the difficulty of reassembling isolated fragments into a coherent class.
• Top-Down for Strong Models: Stronger models like Gemini-2.5-Pro benefited slightly from Top-Down planning, suggesting that explicit high-level planning aids global consistency.

C. Error Analysis (Based on 500 Manually Annotated Failures)

• Dominant Errors:
  • Logic Errors (56.2%): Semantically incorrect implementations (e.g., missing edge cases, wrong exception types).
  • Dependency Errors (38.0%): Failures in managing shared state, method signatures, or imports (e.g., missing import os).
  • Syntax Errors: Negligible (0.6%).
• Core Bottleneck: The primary failure mode is not code formation but cross-method coordination. Models often generate individually correct methods that fail to work together due to inconsistent state management or broken interfaces.

5. Key Contributions

1. ClassEval-Pro Benchmark: A scalable, cross-domain benchmark of 300 tasks designed specifically to evaluate compositional code creation, moving beyond isolated function synthesis.
2. Automated Construction Pipeline: A novel, contamination-resistant pipeline that combines AST-based code merging, LLM Judge Ensembles, and strict execution verification to generate high-quality, real-world tasks without manual curation.
3. Diagnostic Insights: Identification of cross-method coordination as the primary bottleneck in class-level generation. The study reveals that while models can write local methods, they struggle with global class consistency, shared state, and dependency management.

6. Significance

• Realism: By using post-2025 GitHub code and cross-domain composition, ClassEval-Pro offers a more realistic evaluation of LLM capabilities for software engineering tasks than synthetic function-level benchmarks.
• Future Directions: The findings suggest that future LLM improvements for code generation must focus less on syntax and more on architectural reasoning, state management, and dependency coordination.
• Strategy Guidance: The results indicate that generation strategies should be tailored to model capabilities; structured decomposition helps weaker models, while holistic or top-down approaches may suit stronger models better.

The authors have open-sourced the benchmark, pipeline, and evaluation scripts to facilitate further research in this underserved area of code generation.