📄 other

A cross-validated prevalence dataset of 87 cosmetic active ingredients from four public sources

This paper presents a cross-validated dataset of 87 cosmetic active ingredients derived from four diverse public sources, featuring a high-precision rule-based matching ontology and demonstrating strong cross-channel agreement to support reproducible research on cosmetic formulation structures and ingredient co-occurrence.

Original authors: Lejian Wang

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Lejian Wang

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine walking into a giant, chaotic library where every book is a bottle of face cream, shampoo, or lipstick. In this library, the shelves are organized by different rules: one section is filled with books written by enthusiastic fans who wrote down every ingredient they could find; another section is a high-end boutique where only the "clean" and "premium" books are displayed; a third is a massive warehouse listing everything from discount stores to online giants; and a fourth is a strict government filing cabinet that only lists chemicals known to be potentially dangerous. For years, scientists trying to understand what's actually inside our beauty products have had to pick just one section of this library and guess that it represents the whole world. But what if the fan section is full of weird, niche recipes, while the boutique section is hiding the common stuff? To get a true picture of the "cosmic recipe" of beauty, we need to cross-check all these different sections to see if they tell the same story. This is the challenge of studying cosmetic ingredients: figuring out which components are truly universal staples and which are just local trends, without getting tricked by the fact that some books might have been copied from one shelf to another.

This paper is like a master detective who decides to open all four sections of that library at once and compare their contents. The author, Lejian Wang, didn't just guess; they built a super-smart, rule-based robot scanner (a "matcher") that can read the ingredient lists from these four very different sources and translate them into a common language. They focused on 87 specific "active" ingredients—the heavy hitters like moisturizers, anti-agers, and preservatives—that appear in the most popular products. The robot was tested on 50 real products and got it right almost every single time, with a precision of 100% and a recall of nearly 99.6%, meaning it rarely missed a real ingredient and never falsely accused a product of having one.

The big discovery? When the author compared the rankings of these 87 ingredients across the different libraries, the results were surprisingly consistent. Even though the "fan library" (Open Beauty Facts) and the "boutique library" (Sephora) have different vibes and different customers, they agree on which ingredients are the most popular. The math shows a strong connection between them, proving that the popularity of ingredients like glycerin or hyaluronic acid isn't just a fluke of one specific store or country; it's a real, global pattern. The author also did a clever trick to make sure this agreement wasn't just because some products were listed in both the boutique and the warehouse sections; they removed the shared products and the agreement actually got stronger, confirming the signal is real.

However, the paper also draws a sharp line in the sand. When they compared these beauty product lists against the strict government "danger list" (the California Safe Cosmetics Program), the agreement vanished. The government list only cares about a few specific chemicals, so it has no idea what the beauty world is actually using. This lack of agreement is actually a good thing—it proves the robot isn't just hallucinating connections where none exist. The study also peeked into the future by looking at how ingredient popularity has changed over the last decade. It found that some ingredients, like hyaluronic acid, are climbing the charts, while others, like certain sulfates, are fading away. But the author is careful to note that some of these trends, specifically for niacinamide and phenoxyethanol, might be a bit shaky because they are heavily influenced by a few recent years of data, so we should treat those specific trends as exciting hints rather than absolute facts.

In the end, this paper doesn't just give us a list of ingredients; it gives us a verified, cross-checked map of the cosmetic world. It proves that we can trust our understanding of what goes into our products, even when we look at them from different angles, as long as we have the right tools to translate the data. It's a solid foundation for anyone who wants to understand the science behind the bottle, showing that while the world of beauty is vast and varied, the core recipes are more similar than we might have thought.

Technical Summary: A Cross-Validated Prevalence Dataset of 87 Cosmetic Active Ingredients

Problem Statement
Historical studies on cosmetic ingredient co-occurrence and formulation evolution have relied on single, often skewed data sources (e.g., a single crowd-sourced library or one retailer's catalogue). This reliance creates two primary limitations: (1) prevalence rankings derived from a single population cannot be assumed to generalize across geographies or retail channels; and (2) without independent replication, it is impossible to distinguish genuine formulation regularities from noise or inadvertent dataset overlaps. Existing work in food science and traditional medicine utilizes co-occurrence logic but remains confined to single domains or non-INCI lists, while recent AI/ML approaches rely on proprietary datasets that do not validate ranking generalizability across populations.

Methodology
The study constructs a reusable, cross-validated empirical resource by integrating four public datasets representing two distinct "ecological niches" and a regulatory negative control:

EU-leaning Crowd-Sourced Corpus: Open Beauty Facts (OBF, n = 19,045 products).
North-American Retail Niche:
- Premium catalogue: Sephora (n = 1,472).
- Multi-platform e-commerce catalogue: Aggregated from Ulta, Sephora, Amazon, and Flipkart (n = 6,600 products with INCI text).
Regulatory Negative Control: California Safe Cosmetics Program (CDPH, n = 34,502 product-level records), which declares only restricted chemicals rather than full formulations.

Data Processing and Ontology

87-Concept Ontology: A curated dictionary mapping 87 functional concepts (e.g., moisturizers, brighteners, preservatives) to INCI, common, and Latin aliases. The ontology is bounded to ensure every pattern can be hand-authored and audited.
Matching Pipeline: A rule-based matcher (matchIngredientList) applies text normalization (NFD diacritic stripping, boundary regex) and exclusion lists (e.g., preventing "alcohol" from matching "benzyl alcohol") to flag concept presence.
Validation Framework:
- Gold Standard: A stratified random sample of 50 products was manually annotated. The matcher achieved Precision 1.00 and Recall 0.996 (F1 0.998) after repairing three specific defects (false positives on substrings, false negatives due to delimiter issues, and boundary errors).
- Coverage Check: A targeted supplementary sample of 15 products confirmed that all 86 concepts present in the corpora were correctly flagged.
- Statistical Approach: Cross-source agreement is measured using Spearman rank correlation on prevalence vectors. The study prioritizes a "common filter" (concepts with ≥2% prevalence in at least one source) to avoid inflation from shared zeros, while reporting full-87 coefficients as an upper bound. Significance is determined via permutation tests (2,000 iterations) rather than asymptotic distributions.

Key Results

Cross-Source Agreement: Strong rank agreement was observed among the three formulation sources. Under the common filter, Spearman correlations ( $\rho$ $ρ$ ) were:
- OBF $\times$ Sephora: 0.56
- OBF $\times$ E-commerce: 0.65
- Sephora $\times$ E-commerce: 0.81
- Robustness Check: Removing the 2,731 products shared between Sephora and the e-commerce catalogue increased the Sephora $\times$ E-commerce correlation to 0.92, confirming the agreement is not an artifact of data overlap.
Negative Control Validation: The CDPH registry showed negligible correlation with formulation sources ( $\rho$ ranging from -0.03 to 0.13), confirming the pipeline does not manufacture spurious agreement with unrelated regulatory modalities.
Prevalence Patterns: High-prevalence ingredients (e.g., glycerin, tocopherol, phenoxyethanol) showed consistent rankings across sources. Notable divergences, such as the low prevalence of SLES in premium channels (Sephora: 0.9%) versus crowd-sourced data (OBF: 14.9%), reflect channel-specific positioning rather than data error.
Temporal Trends (Exploratory): Analysis of OBF ingestion years (2016–2026) indicated significant upward trends for hyaluronic acid and niacinamide, and downward trends for SLES and dimethicone. However, sensitivity analysis restricted to 2016–2023 revealed that trends for niacinamide and phenoxyethanol were driven largely by the partial 2024–2026 data, rendering them suggestive rather than established.

Significance and Claims
The paper explicitly claims not to have collected new raw data or proposed new statistical methods. Instead, its contribution is a new, openly licensed, cross-validated empirical resource that renders disparate public repositories commensurable.

The significance lies in the validated mapping layer (the ontology and pipeline) which enables:

Generalization Evidence: Demonstrating that functional ingredient rankings generalize across distinct ecological niches (EU crowd-sourced vs. NA retail).
Noise Reduction: Proving that observed agreements are not artifacts of dataset overlap or single-source noise, validated by the negative control and robustness checks.
Reproducibility: Providing a complete, auditable pipeline (scripts, gold-standard samples, and statistics) that allows future researchers to apply the same 87-concept ontology to new corpora.

The author emphasizes that the dataset supports reproducible research on cosmetic ingredient co-occurrence and formulation structure but caution that absolute prevalence levels are not directly comparable across sources due to differing declaration regimes; only rank orders are comparable.

Technical Summary: A Cross-Validated Prevalence Dataset of 87 Cosmetic Active Ingredients

More like this