A Large-Scale Measurement of AI Bill of Materials Completeness in Hugging Face Models
This paper empirically evaluates the completeness of AI Bills of Materials (AIBOMs) across approximately 97,500 Hugging Face models, finding that while required structural fields are fully present, critical AI-specific documentation regarding model cards, licenses, limitations, and safety risks remains significantly incomplete.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
Imagine you just bought a delicious, pre-made sandwich from a popular food truck. It looks great, but the wrapper is blank. You have no idea what ingredients are inside, where the bread was baked, who made the filling, or if it contains nuts that might make you sick. In the world of artificial intelligence, developers often do exactly this: they grab powerful, pre-trained AI models from massive online libraries to build their own apps, without knowing the "ingredients" or the "recipe." This is risky because if the AI makes a mistake or acts strangely later, no one knows why. To fix this, scientists are trying to create something called an "AI Bill of Materials" (AIBOM). Think of an AIBOM as a digital nutrition label or a detailed receipt for an AI model. It's a structured list that tells you everything about the model: what it was trained on, who made it, what rules (licenses) you have to follow, and what its weaknesses are. The big question is: if we automatically generate these "nutrition labels" for the millions of AI models out there, do they actually tell us the truth, or are they mostly empty wrappers?
This paper takes a giant bite out of that question by investigating the "nutrition labels" of nearly 98,000 AI models hosted on Hugging Face, the internet's biggest marketplace for AI models. The researchers used a special tool to automatically scan these models and generate their AIBOMs, then checked how complete and useful those labels were. They found that while the labels are technically valid—meaning they have the right structure and basic ID tags—they are surprisingly empty when it comes to the important stuff. It's like having a receipt that lists the price and the store name but leaves the ingredients, the allergen warnings, and the expiration date completely blank.
The study reveals that the generated AIBOMs are "moderately complete" with an average score of about 54 out of 100. The good news is that the basic skeleton of the document is always there; the tool successfully identifies the model's name, version, and format 100% of the time. However, the "meat" of the document is missing. Critical details like the specific datasets used for training, safety warnings, ethical limitations, and environmental impact information are largely absent. For instance, while the tool found that about 39% of models listed their training datasets, a staggering 0% of the generated labels included information about "ethical considerations" or "intended use." Even the "meaningful descriptions" of what the model actually does were missing in 99.8% of cases, often replaced by blank placeholders.
The researchers also discovered that the quality of these labels depends heavily on how well the original model was documented. Models that came with a research paper or a clear statement about their training data tended to get much better AIBOMs, with more links to code and safety checks. But for the vast majority of models, especially those without these extra signals, the generated labels remain weak. The study suggests that while we have the technology to create these transparency documents, the AI community isn't yet providing enough information to fill them out. Until model creators start treating documentation as a necessary part of their product—like a chef listing ingredients on a menu—these AI "nutrition labels" will remain incomplete, leaving developers and users in the dark about what they are actually using.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.