Technical Summary: CodeXHug – A Dataset to Enhance Model Cards with Code Usage Patterns

1. Problem Statement

Pre-trained models (PTMs) have become integral to software engineering, facilitated by repositories like Hugging Face (HF). However, the actual adoption of these models in real-world projects remains an open question, with many instances of usage limited to "toy projects" or mere mirrors of the HF repository. A critical barrier to adoption is the lack of concrete, explanatory code patterns in existing model cards. While model cards provide metadata and installation instructions, they often fail to include practical usage examples, making integration challenging for non-expert developers. Furthermore, existing documentation frequently lacks critical information such as license details or carbon emission data. Identifying useful code snippets for specific PTMs within the vast GitHub ecosystem is difficult due to the prevalence of low-quality or irrelevant repositories.

2. Methodology

The authors propose CodeXHug, a curated dataset linking Hugging Face PTMs to their actual usage patterns in GitHub repositories. The data collection process involves four primary stages:

1. Data Cleaning and Sampling:

   • The process begins with the latest Hugging Face dump (June 2024), containing 681,682 models.
   • Models with null content (missing tags or model cards) are filtered out, reducing the set to 262,670 PTMs.
   • A sample of the top 10% most popular models based on download counts is selected (20,545 PTMs) to ensure the dataset reflects real-world usage likelihood.

2. Tag Filtering:

   • To ensure diversity, the authors identify 13 representative pipeline categories (e.g., text-generation, fill-mask, image-classification).
   • This step balances the dataset by including both highly popular and less popular tags, resulting in 17,760 PTMs.

3. GitHub Mapping:

   • Using the PyGithub library, the authors query the GitHub platform for repositories written in Python that reference the selected PTMs.
   • Queries are refined by checking README content for pipeline tags to reduce false positives.
   • For each identified project, the authors collect up to 1,000 Python source files (.py) and the project's README content.
   • Metadata such as commits, pull requests, forks, and stars are also recorded.

4. Data Storage:

   • The final dataset is stored in a MongoDB database to facilitate faster access compared to SQL. It comprises two collections: models (metadata) and files (source code content encoded in UTF-8).

5. Usage Pattern Extraction (Demonstration):

   • To demonstrate the dataset's utility, the authors apply a statistical analysis and clustering technique to extract representative code usage patterns.
   • Outlier Filtering: Models with fewer than 100 associated files are filtered out to focus on well-used models, reducing the set to 1,064 models.
   • Snippet Filtering: Code snippets are filtered based on length (using Interquartile Range centered on the median) to exclude trivial scripts or overly long documentation-heavy files.
   • Clustering: Snippets are converted into 384-dimensional vectors using the all-MiniLM-L6-v2 embedding model. The K-means algorithm groups these vectors to identify semantic similarities.
   • Generation: The snippet closest to each cluster centroid is selected as a representative exemplar. These exemplars are then fed into the Llama 3 model using a zero-shot prompt to generate structured code usage patterns (e.g., data loading, preprocessing, model definition).

3. Key Contributions

The paper presents three primary contributions:

• CodeXHug Dataset: A curated dataset mapping 7,325 distinct Hugging Face PTMs to 372,063 Python files across 71,748 GitHub repositories. This is the first dataset to provide a comprehensive overview of actual PTM usage in the GitHub ecosystem.
• Qualitative Analysis: An analysis of the collected code snippets to provide developers with concrete usage examples, addressing the gap in current model card documentation.
• Usage Scenario: A proof-of-concept demonstrating how to predict and generate representative code usage patterns for specific PTMs by combining statistical filtering, K-means clustering, and Large Language Models (LLMs).

4. Results and Observations

• Dataset Scale: The final dataset contains 7,325 unique models and 372,063 Python files.
• Distribution Imbalance: The file distribution across pipeline tags is unbalanced. text-generation is the most dominant tag, followed by fill-mask and image-classification, reflecting the popularity of models like GPT-2 and RoBERTa.
• Usage Patterns: The analysis reveals a "long-tail" effect in PTM popularity. A small number of models (e.g., gpt-2, roberta-large, Salesforce/blip-image-captioning-large) are used in hundreds of projects (over 800), while the majority are used in very few.
• Multi-Model Usage: Approximately 43.7% of the analyzed repositories utilize more than one PTM, suggesting that developers often combine models to support complex tasks.
• Pattern Generation: The Llama 3-based approach successfully generated coherent code snippets (e.g., for the kakaobrain/karlo-v1-alpha-image-variations model) that included standard procedures for data loading, preprocessing, and model architecture definition, despite the zero-shot prompting paradigm.

5. Significance and Claims

The authors position CodeXHug as a foundational resource for the software engineering community to bridge the gap between theoretical model availability and practical implementation. The paper claims that CodeXHug can support:

• Enhanced Model Cards: Automatically generating model cards enriched with concrete code usage examples.
• Developer Support: Providing API recommendations and code usage pattern retrieval for developers integrating PTMs.
• Research Opportunities: Enabling studies on code quality (e.g., detecting code smells or technical debt in AI systems), the impact of PTM adoption on project popularity, and the development of next-generation code assistants.

The authors maintain a modest stance, acknowledging that the current dataset focuses on popular models and that the clustering and generation techniques presented are illustrative. They note that further validation and the exploration of more advanced clustering or different LLMs are necessary for broader generalizability. The dataset is made available via the Zenodo open-access repository.