🤖 AI

Industry Classification of GitHub Repositories Using the North American Industry Classification System (NAICS)

This paper introduces NAICS-GH, a high-precision, publicly released corpus of 6,588 GitHub repositories labeled with NAICS 2022 industry sectors using a retrieve-and-verify pipeline combining BAAI/bge-large-en embeddings and GPT-4.1, which achieves 96.98% human-validated precision and serves as a benchmark for industry classification in open-source innovation research.

Original authors: Kevin Xu, Alexander Quispe

Published 2026-07-08

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Kevin Xu, Alexander Quispe

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine GitHub as a massive, chaotic library containing hundreds of millions of books (code repositories). The problem is that while the library knows the title of every book, it has no "Dewey Decimal System" to tell you what kind of business or industry that book is actually for. Is this code for a bank? A farm? A video game studio? Without this label, it's hard for economists, policymakers, or companies to understand how different industries are using open-source software.

This paper introduces NAICS-GH, a new system that acts like a super-smart librarian to organize these books into the North American Industry Classification System (NAICS)—the standard filing system used by governments in the US, Canada, and Mexico.

Here is how they built it, explained simply:

1. The "Fishing Net" (Retrieval)

First, the team couldn't read every single book in the library (there are over 1.3 million candidates). So, they used a digital fishing net called BGE embeddings and FAISS.

The Metaphor: Imagine you have a list of 1,000 specific keywords (like "crop yield," "banking security," or "hospital scheduling"). You throw these keywords into the library, and the net instantly grabs the top 20 books that sound most similar to each keyword.
The Result: This net caught about 31,000 potential matches. It cast a wide net to make sure they didn't miss anything, but it also grabbed some "near-misses" (books that sounded similar but weren't quite right).

2. The "Expert Judge" (Verification)

The net was too loose, so they needed a strict judge to verify the catches. They hired GPT-4.1 (a very advanced AI) to act as a domain expert.

The Metaphor: Think of GPT-4.1 as a seasoned inspector. For every book the net caught, the inspector reads the cover, the summary, and the first few pages (the repository's description and README).
The Rubric: The inspector doesn't just guess; they use a strict checklist (a "rubric"). They ask: "Does this code actually solve a problem for this specific industry?"
- If the answer is a clear "Yes," the inspector gives it a score of 9 or 10.
- If it's a "Maybe," the score is lower.
- If it's a generic tool used by everyone (like a basic calculator), it gets a low score.
The Cut-off: They only kept the books that got a score of 8 or higher. This ensured they only kept high-confidence matches.

3. The Final Collection (The Dataset)

After the AI judge filtered out the weak matches, they were left with 6,588 high-quality repositories.

These are spread across 19 different industries (like Agriculture, Manufacturing, Healthcare, and Finance).
They intentionally left out one category ("Management of Companies") because there weren't enough clear examples to make a reliable group.
Crucially: They removed the names of the owners to protect privacy, keeping only the code content and the industry label.

4. Did it Work? (Validation)

To make sure the AI judge wasn't hallucinating, the team hired human research assistants to randomly check 2,421 of these labeled books.

The Result: The human judges agreed with the AI 96.98% of the time.
The Nuance: The AI was almost perfect for clear-cut industries like "Public Administration" or "Arts & Entertainment." However, it struggled a bit more with "Manufacturing" and "Wholesale Trade," where software can sometimes look generic. The paper notes that if you raise the score requirement to 9 or 10, the accuracy in those tricky industries goes up even higher.

5. The "Teaching Tool" (Benchmark)

Finally, the team used this new, labeled library to teach six different AI models how to classify code on their own.

They tested models like RoBERTa, ModernBERT, and DeBERTa.
The Winner: A model called RoBERTa-large learned the best, achieving 86.45% accuracy on a test set it had never seen before.
The Takeaway: This proves that once you have a good "labeled library," you can train smaller, faster AI models to do the classification work automatically in the future.

Summary

The paper presents a publicly available dataset that maps GitHub code to real-world industries. It was built by:

Casting a wide net to find potential matches.
Using a strict AI judge to verify them against a detailed checklist.
Having humans double-check the work to ensure 97% accuracy.
Releasing the data and the code so anyone can study how different industries use open-source software.

The authors explicitly state this is a North American classification system applied globally, and they warn that while the labels are highly accurate, they are not perfect for every single industry (specifically manufacturing and wholesale). They also note that the system currently works best with English-language code descriptions.

Technical Summary: Industry Classification of GitHub Repositories Using NAICS

Problem Statement
GitHub hosts hundreds of millions of public repositories but lacks a native mechanism to map these repositories to standardized industry sectors. This absence of structured industry metadata limits empirical research into the geography of innovation, the industrial composition of open-source production, and the diffusion of technologies across economic sectors. Existing classification efforts have relied on user-declared topic tags or software application domains, but none have adopted a standardized economic taxonomy like the North American Industry Classification System (NAICS) across multiple jurisdictions.

Methodology: The NAICS-GH Pipeline
The authors present NAICS-GH, a corpus of 6,588 GitHub repositories labeled with 2-digit NAICS codes (NAICS 2022). The dataset is constructed using a reproducible, two-stage "retrieve-and-verify" pipeline applied to source pools from the United States, the European Union, and Australia.

Source Pool Extraction:
- Data: 1,372,489 public, non-fork repositories were extracted from GitHub's internal data warehouse.
- Inclusion Criteria: Repositories must be public, have at least one star, contain a README of ≥750 bytes, have >6 commits, and involve ≥2 distinct contributors.
- Jurisdictions: The pool is split by owner account country: USA (510,380), EU-27 (530,898), and Australia (331,211).
Stage 1: Semantic Retrieval (Candidate Generation):
- Embedding: Repository READMEs (first 1,000 characters) are encoded using BAAI/bge-large-en embeddings.
- Query Construction: The pipeline uses a taxonomy of 1,029 distinct 6-digit NAICS subindustry phrases. For each phrase, a query "Repositories about {subindustry}" is generated.
- Retrieval: Using FAISS (IndexFlatIP) for exact inner-product search, the system retrieves the top- $k$ most semantically similar repositories. The $k$ value is adaptive: $k=20$ for sectors with many subindustries, and boosted ( $k \approx 24\text{--}29$ ) for sectors with few subindustries to ensure roughly 400 candidates per sector.
- Output: This step yields 31,178 unique (repository, sector) candidate pairs.
Stage 2: LLM Verification (Filtering):
- Model: GPT-4.1 (snapshot gpt-4.1-2025-04-14) is used as a verifier.
- Input: The model receives a structured prompt containing the repository's description, topic tags, and cleaned README (concatenated), alongside the target NAICS sector definition.
- Rubric: The model scores the match on a 1–10 scale based on four criteria: industry-specific software, sector-relevant functionality, industry domain applications, and sector-specific data/research.
- Thresholding: Candidates with a score $\ge 8$ are retained. A minimum class-size filter ( $n \ge 80$ ) is applied, removing Sector 55 ("Management of Companies and Enterprises") which only had 13 passing candidates.

Key Contributions

Dataset (NAICS-GH): A publicly released corpus of 6,588 repositories labeled with 2-digit NAICS codes, covering 19 of the 20 NAICS sectors. The dataset includes repository metadata, cleaned text, and the LLM's confidence score and rationale.
Reproducible Pipeline: A complete, open-source pipeline (MIT License) combining dense retrieval (BGE + FAISS) with LLM verification (GPT-4.1). The pipeline is designed to be adaptable to other industry taxonomies.
Validation: A manually verified "gold" subset of 2,421 repositories confirms the pipeline's precision.
Benchmarks: A head-to-head comparison of six pretrained encoders (RoBERTa, ModernBERT, DeBERTa-v3) fine-tuned on the corpus.

Results and Validation

Precision: On the human-validated gold sample, the pipeline achieves 96.98% precision (95% Wilson CI: [96.23%, 97.59%]).
Score Correlation: Precision correlates monotonically with the GPT-4.1 score: 90.76% at score 8, 97.81% at score 9, and 99.30% at score 10.
Sector Variance: Six sectors (including Public Administration, Information, and Mining) achieved 100% precision in the gold sample. However, Manufacturing (31–33) and Wholesale Trade (42) showed lower precision (73.0%) at the score $\ge 8$ threshold, suggesting these sectors are harder to distinguish from generic supply-chain tooling.
Reproducibility: Re-running the pipeline on different hardware reproduces the candidate set to within 0.03% of the original size.
Benchmark Performance: Among six fine-tuned encoders, RoBERTa-large achieved the highest test-set performance with 86.45% F1 and 86.35% accuracy. Notably, larger parameter counts did not uniformly improve performance (e.g., ModernBERT-large underperformed ModernBERT-base), suggesting that for this corpus size (~4.6k training examples), pretraining objectives matter more than scale.

Significance and Claims
The paper claims to provide the first publicly available, multi-jurisdictional corpus mapping GitHub repositories to the NAICS standard. This resource addresses a critical gap for economists, policymakers, and researchers studying the software sector's labor and capital allocation. The authors position the dataset not as a perfect ground truth (acknowledging label noise in specific sectors and the English-only limitation of the retrieval model) but as a high-confidence resource that enables empirical work on the industrial composition of open source. The release includes the dataset, fine-tuned checkpoints, and full pipeline code to facilitate downstream research and replication.