OpenClassGen: A Large-Scale Corpus of Real-World Python Classes for LLM Research

1. Problem Statement

Current benchmarks for Large Language Model (LLM) code generation face a significant gap between scale and realism, particularly at the class level:

• Function-level benchmarks (e.g., HumanEval, MBPP) are too simple, failing to capture the structural complexity of object-oriented design (inheritance, coupling, state management).
• Repository-level benchmarks (e.g., RepoClassBench) require complex agentic architectures to navigate cross-file dependencies, making evaluation costly and difficult to scale.
• Existing Class-level benchmarks are insufficient:
  • ClassEval: Synthetic, small scale (100 classes), and lacks real-world variability.
  • RealClassEval: Real-world but too small (400 classes) for training, fine-tuning, or statistically robust empirical analysis.

There is a critical need for a large-scale, real-world dataset that isolates the class-level generation task (using self-contained skeletons) without requiring full repository context, enabling both training and rigorous evaluation.

2. Methodology

The authors present OpenClassGen, a curation pipeline designed to extract and structure Python classes from production-grade open-source projects.

A. Data Collection & Filtering

• Source: 2,970 "engineered" open-source GitHub repositories selected based on strict criteria: valid licenses, active maintenance (releases, contributors), and high quality (PRs, issues, code-to-comment ratios).
• Extraction:
  • Cloned repositories and parsed all .py files using Python's ast module.
  • Generated Class Skeletons: These retain class/method signatures and docstrings but replace method bodies with pass statements. This serves as the generation prompt.
  • Paired each skeleton with the original Human-Written Implementation.
• Filtering:
  • Removed classes failing AST parsing (e.g., Python 2 syntax).
  • Removed test classes (identified by naming conventions like Test or file paths containing test) to focus on production code generation.
• Final Corpus: 324,843 production Python classes.

B. Enrichment

• Static Metrics: Each class is annotated with 27 static code metrics covering:
  • Size: Lines of code, methods count.
  • Complexity: Cyclomatic complexity, nesting depth.
  • Coupling: Fan-in, fan-out, dependencies.
  • Inheritance: Depth of tree, children count.
• Documentation Analysis: Classes are categorized by documentation coverage (Fully documented, Partially documented, Undocumented).

C. Evaluation Strategy

To demonstrate utility, the authors evaluated three LLMs (GPT-o4-mini, Claude-4-Sonnet, Qwen-3-Coder) on a stratified random sample of 300 classes (100 per documentation tier).

• Prompting: Models were asked to implement the full class based only on the skeleton.
• Metrics:
  1. Syntactic: BLEU and ROUGE-L (lexical overlap).
  2. Semantic: CodeBERTScore-F3 (embedding similarity).
  3. Structural: Tree Similarity Edit Distance (TSED) comparing Abstract Syntax Trees (ASTs).
  4. Functional: Pass Rate on automatically generated unit tests using Pynguin (achieving 58% branch coverage against ground truth).

3. Key Contributions

1. OpenClassGen Corpus: The first large-scale (324k+ classes) real-world Python dataset specifically curated for class-level code generation research.
2. Rigorous Curation Pipeline: A reproducible process involving engineered project filtering, AST-based skeleton extraction, and test-class removal.
3. Rich Metadata: Inclusion of 27 static metrics per class, enabling research into difficulty modeling, structural analysis, and retrieval-augmented generation (RAG).
4. Empirical Validation: A multi-dimensional evaluation showing that the dataset effectively differentiates model capabilities across syntactic, semantic, structural, and functional dimensions.
5. Public Release: Full dataset and curation scripts are publicly available via Zenodo and Hugging Face.

4. Key Results

The evaluation of the 300-class subset yielded the following insights:

• Semantic & Structural Strength: Models achieved high semantic similarity (CodeBERTScore-F3: 0.89) and structural alignment (TSED: 0.83), indicating they understand class design and intent.
• Syntactic Variation: Moderate lexical overlap (BLEU: 0.48, ROUGE-L: 0.65), suggesting models generate valid but stylistically different implementations.
• Functional Gap: The Pass Rate was low (0.33 average), with significant variance between models (GPT-o4-mini: 0.29, Claude-4-Sonnet: 0.36, Qwen-3-Coder: 0.35).
• Differentiation: The substantial variance in pass rates confirms that OpenClassGen is sensitive enough to distinguish between different model architectures and capabilities.
• Implication: There is a distinct gap between understanding a class specification (high semantic score) and executing it correctly (low pass rate), highlighting a key challenge for future research.

5. Significance and Future Impact

OpenClassGen addresses the "scale-realism trade-off" in code generation research:

• Training & Fine-tuning: The scale (324k pairs) is sufficient for supervised fine-tuning of models on class-level tasks, which was previously impossible with datasets like RealClassEval.
• RAG & Difficulty Modeling: The 27 static metrics allow researchers to build retrieval systems based on structural similarity (e.g., finding classes with similar complexity) and to predict generation difficulty based on code metrics.
• Failure Analysis: The dataset enables statistically robust analysis of LLM failure modes (e.g., correlating high coupling with specific error types).
• Documentation Studies: The natural variation in documentation (20% full, 44% partial, 36% none) allows for realistic studies on how documentation quality impacts generation performance.

In summary, OpenClassGen transforms class-level code generation from a niche, small-scale evaluation task into a scalable research infrastructure, bridging the gap between isolated function generation and complex repository-level synthesis.