💬 NLP

Building a Custom Taxonomy of AI Skills and Tasks from the Ground Up with Job Postings

This paper introduces TaxonomyBuilder, a framework demonstrating that filtering job posting data before processing yields clearer and more domain-specific AI skill taxonomies than using unfiltered corpora with LLM-enhanced clustering tools.

Original authors: Stephen Meisenbacher, Peter Norlander

Published 2026-05-21

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Stephen Meisenbacher, Peter Norlander

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to organize a massive, chaotic library containing millions of books about Artificial Intelligence (AI). Your goal is to create a clear, easy-to-read map (a taxonomy) that helps people find exactly what they need: specific AI skills and tasks.

The authors of this paper, Stephen Meisenbacher and Peter Norlander, built a new tool called TAXONOMYBUILDER to do this job automatically. They tested it using two giant piles of real-world data: millions of job postings from the US labor market.

Here is the story of what they found, explained simply:

The Problem: The "More is Better" Trap

Usually, when people try to organize a huge mess, their instinct is to throw everything into the mix. They think, "If I include more job postings, my map will be more complete." They might even try to copy-paste similar sentences to make the pile even bigger (a process called data augmentation).

The authors asked: Is this actually helpful? Or does it just make the library messier?

The Experiment: Cooking with Different Ingredients

To find the answer, they ran 24 different experiments using their TAXONOMYBUILDER tool. They changed three main "ingredients" in their recipe:

Data Augmentation (Adding more): Did they add extra, similar sentences to the mix to make the dataset bigger?
Filtering (Removing the noise): Did they throw away the "weakest" or most vague job descriptions, keeping only the top 25%, 50%, or 75% of the clearest ones?
Soft Clustering (Fuzzy grouping): Did they force "noisy" items that didn't quite fit into a group to join anyway, just to be safe?

The Big Surprise: Less is More

The results were counter-intuitive. The team discovered that adding more data actually made the map worse.

The "Augmentation" Mistake: When they added extra data to make the pile bigger, the resulting map became confusing and less accurate. It was like trying to organize a library by adding more books that were just slightly different copies of the same story; it just created clutter.
The "Filtering" Win: The best maps were created when they were strict. They threw away the vague, low-quality job descriptions (keeping only the top 75% or 50% of the best data). By removing the "noise," the AI could see the clear patterns much better.
The "Soft Clustering" Nuance: Forcing messy items into groups only helped if they hadn't added extra data in the first place. If they added extra data, forcing messy items in made things worse.

The Analogy: The Gold Panning Test

Imagine you are panning for gold (AI skills) in a massive river (the job postings).

The Old Way: You scoop up every bucket of water, mud, and rocks you can find, hoping to get more gold. You end up with a huge, muddy pile where it's impossible to see the gold.
The New Way (TAXONOMYBUILDER's finding): You carefully sift through the water first. You throw away the muddy, useless buckets (filtering). You only keep the buckets that look promising. Then, you pan those. You end up with a smaller pile, but it is pure gold.

The Conclusion

The paper concludes that when building a map of complex skills from massive amounts of text, quality beats quantity.

Don't try to include every single piece of data you can find.
Do be selective. Filter out the weak or noisy information.
Do let the AI focus on the clear, high-quality examples to build a map that is actually useful and easy to understand.

In short: If you want a clear map of the AI job market, you don't need to read every single job posting in existence. You just need to read the best ones carefully.

Technical Summary: Building a Custom Taxonomy of AI Skills and Tasks from the Ground Up with Job Postings

Problem Statement
The rapid evolution of the labor market, particularly regarding Artificial Intelligence (AI), necessitates the creation of open, granular taxonomies to track changes in skills and tasks. While proprietary taxonomies exist, they often suffer from data access restrictions and a lack of standardization, limiting their utility for public goods such as workforce development and educational design. Although Large Language Models (LLMs) offer a pathway to automate taxonomy construction from massive text corpora (e.g., job postings), a critical gap remains in understanding how to best leverage such large-scale data. Specifically, it is unclear whether including more data (via augmentation) or filtering data strictly yields better domain-specific coverage and clarity. Previous works have focused on smaller datasets or relied on seed taxonomies, leaving the systematic evaluation of design parameters for massive, unstructured corpora unexplored.

Methodology: TAXONOMYBUILDER
The authors propose TAXONOMYBUILDER, a systematic pipeline designed to transform large-scale, unstructured job posting text into hierarchical taxonomies. The method operates in three primary stages:

Setup and Context Mining:
- Candidate Extraction: The system performs a keyword-based search using a curated list of 590 AI-related keywords to extract candidate sentences from massive job posting corpora.
- Scoring and Filtering: To mitigate false positives (e.g., "torch" as a welding tool vs. a library), candidates are scored using a hybrid method based on cosine similarity between sentence embeddings and keyword embeddings. Three embedding models (ALL-MINILM-L12-V2, GTE-LARGE, EMBEDDINGGEMMA-300M) are averaged for stability.
- Configuration Variables: The study investigates three key design decisions:
  - Data Augmentation: Adding semantically similar sentences from the corpus to the candidate set (threshold > 0.9).
  - Percentile Filtering: Retaining only the top 25th, 50th, or 75th percentile of scored candidates to reduce noise.
  - Soft Clustering: Re-assigning "noise" points (those not fitting a cluster) to the nearest cluster centroid.
Forming the Foundation (Leaf Level):
- Clustering: Candidate sentences are clustered using HDBSCAN (with UMAP dimensionality reduction to 10 dimensions).
- Generative Labeling: An LLM (GPT-4o-mini) generates concise labels for each cluster, summarizing the candidate sentences.
- Pruning: Generated labels undergo a similarity-based check; highly similar labels (cosine similarity > 0.95) are aggregated to remove redundancy.
Building Up (Hierarchical Construction):
- The process iterates vertically: the labels from the current level serve as the input for the next level of clustering and labeling.
- The process stops when the number of labels falls below a threshold (default 10) or a maximum depth (default 5) is reached.

Datasets
The pipeline was evaluated on two large US job posting corpora:

NLx: 32.12 million job postings from 2024–2025.
USAJOBS: 1.54 million federal job postings from 2017–2023.
A portion of the most recent data from each corpus was held out as a test set for external validity evaluation.

Evaluation Metrics
The authors employed a multi-faceted evaluation strategy across 24 experimental configurations (12 per dataset):

Clustering Quality: Silhouette score to measure cluster definition.
LLM-as-a-Judge: An LLM (GPT-5.4-Nano) evaluated the taxonomies on four categories: Clarity (precision, ambiguity, consistency, accessibility), Hierarchical Coherence, Orthogonality (distinctiveness, non-overlap), and Completeness (coverage, depth, balance).
Domain Coverage: Measured by how well the resulting taxonomy labels could classify held-out job sentences as AI-related. This included "Lenient" (at least one LLM agrees) and "Strict" (both LLMs agree) coverage rates at various similarity thresholds ( $\tau$ ), alongside a "Label Utilization" metric.

Key Results
The experimental results, analyzed via factorial ANOVA, reveal a counter-intuitive finding: "Less is more."

Data Augmentation: Contrary to the intuition that more data improves coverage, data augmentation significantly reduced strict coverage rates and label utilization. It introduced noise that degraded the quality of the resulting taxonomy.
Filtering: Stricter filtering (higher percentiles, e.g., keeping only the top 25th or 50th percentile of scores) generally improved cluster fit, orthogonality, and label utilization.
Soft Clustering: Soft clustering (re-assigning noise points) was beneficial only when data augmentation was not used. When combined with augmentation, it did not yield significant improvements.
Trade-offs: The study identified an inverse relationship between data inclusion and coverage performance depending on the strictness of the evaluation threshold. While larger, noisier taxonomies (low filtering) performed better at very high similarity thresholds (e.g., $\tau=0.9$ ), more concise, filtered taxonomies outperformed them at lower thresholds (e.g., $\tau=0.6$ ) due to better-defined concepts.

Contributions and Significance
The paper makes four primary contributions:

Systematic Investigation: It addresses the unexplored question of optimal data usage (augmentation vs. filtering) in automated taxonomy construction for massive domains.
TAXONOMYBUILDER: It introduces a generalizable, open-source pipeline for constructing hierarchical taxonomies from unstructured text, demonstrated on the complex domain of AI workplace skills.
Empirical Evidence: It provides clear recommendations that careful augmentation, strict clustering, and selective filtering yield more comprehensive yet concise taxonomies than unfettered data inclusion.
Open Resources: The authors release the constructed AI skills taxonomies and the TAXONOMYBUILDER tool (available as a Python package and on GitHub) to support further research and workforce development.

Conclusion
The authors conclude that while massive data corpora offer a fertile foundation for taxonomy construction, indiscriminate inclusion of data can be detrimental. The study advocates for a "less is more" philosophy, where selective filtering and careful handling of noise are essential for creating high-quality, interpretable, and domain-relevant taxonomies. The work cautions that these findings are case-specific and that future research should continue to explore design trade-offs and formalize evaluation metrics for real-world generalizability.