💬 NLP

Security and Privacy Taxonomy Generation from Mobile App Reviews

This paper introduces TaxoScale, a scalable pipeline that filters over 600,000 mobile app reviews and employs recursive hierarchical clustering combined with LLM-based naming to automatically generate a comprehensive and novel security and privacy taxonomy, overcoming the limitations of existing methods that struggle with large-scale datasets.

Original authors: Moghis Fereidouni, Vinaik Chhetri, Umar Farooq, A. B. Siddique

Published 2026-08-11

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Moghis Fereidouni, Vinaik Chhetri, Umar Farooq, A. B. Siddique

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine the internet as a giant, bustling city where every app is a shop, a restaurant, or a park. Every day, millions of people walk through these digital doors, and when they leave, they often shout their complaints or praise into the air. These shouts are "app reviews." While some people complain about slow loading times or ugly colors, others are screaming about something much more serious: their privacy and security. They are saying, "This app is watching me!" or "It's stealing my data!"

For a long time, scientists and security experts have tried to listen to these shouts to understand what's going wrong. They created "taxonomies," which are like giant, organized filing cabinets or library catalogs. Instead of just hearing a chaotic mess of complaints, a taxonomy sorts them into neat categories like "Data Collection," "Surveillance," or "Password Problems." However, there's a big problem with these old filing cabinets: they were built by hand, one folder at a time, by human experts. The city of apps changes too fast for humans to keep up. New features appear, new tricks are used to spy on users, and the old filing cabinets become dusty and outdated before they are even finished. The question becomes: How do we build a filing system that can organize millions of shouts in real-time, without getting overwhelmed?

This is where the researchers from the University of Kentucky and Louisiana State University step in with a new tool called TaxoScale. They realized that the old way of building these catalogs was too slow and couldn't handle the sheer volume of data. They had a massive pile of 18.63 million app reviews, but only about 601,257 of them were actually about privacy or security. Trying to sort that many complaints by hand would take forever, and trying to use a standard computer program to do it would likely crash because the list is just too long.

To solve this, the team built a smart pipeline that acts like a super-efficient librarian. First, they used a powerful AI (a type of computer brain called an LLM) to act as a filter, sifting through the 18 million reviews to find the 600,000 that were actually about privacy and security. Then, they used another AI to pull out the specific sentences where users described their worries, turning long rants into short, clear "labels" like "needs my contacts" or "tracks my driving."

The real magic happens in the next step. Instead of trying to sort all 600,000 labels at once (which would be like trying to organize a million books in a single room), TaxoScale uses a clever trick called Recursive Hierarchical Clustering. Imagine you have a huge pile of mixed-up toys. Instead of trying to sort them all at once, you first split the pile into two big buckets. Then, you take one bucket, split it into two smaller buckets, and keep doing that until the piles are small enough to sort easily. Once the small piles are sorted, you glue the groups back together in a logical order. This "divide and conquer" method allows the system to handle the massive scale without getting stuck.

Finally, the system uses an AI to give names to these new groups. It looks at the sorted piles and invents clear, short names for them, like "Behavioral Tracking" or "Network Security." The result is a brand-new, living taxonomy that is much better than the old hand-made ones. The researchers found that their new system was not only more accurate at sorting the complaints but also discovered entirely new categories that no one had thought of before. For example, they found a whole new branch of concerns about "Network Security" (like how apps manage IP addresses or use VPNs) and a very specific category for "Parental Control" issues, which makes sense because their data included reviews from teacher-approved apps.

The paper shows that TaxoScale is a significant step forward. It doesn't just copy the old categories; it finds new ones and organizes them better than previous automatic methods. By releasing their data and code, the researchers are handing the keys to this new, smarter filing system to the rest of the world, hoping it will help keep our digital city safer and more transparent as it continues to grow.

Technical Summary: Security and Privacy Taxonomy Generation from Mobile App Reviews

Problem Statement

Mobile app reviews constitute a massive, continuously evolving source of user feedback regarding privacy and security. Unlike static developer-defined policies, these reviews capture real-world user experiences with intrusive permissions, data collection, and surveillance. However, existing taxonomies for organizing these concerns are hand-crafted, static, and expensive to maintain, failing to keep pace with the dynamic nature of app ecosystems.

Automating taxonomy construction is the natural solution, yet current methods face significant scalability barriers. Existing approaches utilizing Large Language Models (LLMs) and clustering are typically designed for scientific corpora of a few thousand documents. They struggle with app review collections that number in the hundreds of thousands (even after filtering) due to LLM context window limitations and the computational intractability of hierarchical clustering on such large datasets. Furthermore, there is a lack of large-scale, open, privacy-specific datasets, as prior resources are either narrow in scope (e.g., health apps only) or not publicly released.

Methodology: TaxoScale

To address these challenges, the authors introduce TaxoScale, a four-step pipeline designed to construct taxonomies at the scale of 600,000+ reviews. The system seeds from an existing expert-defined taxonomy (Akgul et al., 2024) and extends it using a combination of embedding-based clustering and LLM-based node naming.

1. Corpus Construction and Pseudo-Labeling

Data Aggregation: The authors aggregated 18.63 million reviews from three sources (AI-powered reviews, MobileRec, and teacher-approved apps) covering 11,220 apps.
Filtering: An LLM-based classifier (Llama-3.3-70B) filtered these reviews to identify 601,257 privacy- and security-related reviews. The model was selected after benchmarking ten open-source LLMs across eight prompt templates, achieving an F1 score of 0.9700.
Extraction: An extractive summarizer (also Llama-3.3-70B) isolated specific text spans describing concerns within the filtered reviews. These spans serve as "pseudo-labels" (e.g., extracting "need my contacts and call logs" from a review praising the UI but complaining about permissions).

2. The TaxoScale Pipeline

The pipeline proceeds through four distinct stages:

Step 1: Embedding and Assignment: Pseudo-labels are encoded using Qwen3-Embedding-8B and assigned to nodes in the seed taxonomy based on cosine similarity. Labels falling below a similarity threshold are collected as unassignable data to be attached to the root.
Step 2: Recursive Hierarchical Clustering: This is the core innovation for scalability. Instead of applying a single clustering algorithm to the entire dataset, TaxoScale alternates between:
- Top-down partitioning: Using mini-batch k-means to split large clusters.
- Bottom-up merging: Using Ward's agglomerative linkage on the resulting centroids.
- Recursion: If a resulting cluster remains too large (exceeding a tractability threshold $\tau$ ), the process recurses on that sub-cluster. This allows the system to handle hundreds of thousands of labels while remaining grounded in established terminology.
Step 3: Binary-to-Multiway Conversion: The initial clustering produces a binary tree, which is often too fine-grained. The system collapses nodes based on merge height thresholds ( $\delta_{leaf}$ and $\epsilon$ ) to convert the binary structure into a multi-way taxonomy that better reflects human categorization.
Step 4: LLM-Based Node Naming: Newly created nodes are named bottom-up using Qwen3-235B. The model is prompted with the node's pseudo-labels (or child names for internal nodes), the parent path, and sibling names to generate concise (2–6 word) summaries that are distinct from siblings and strictly sub-categorical to the parent.

Key Contributions

Large-Scale Privacy Corpus: The creation of a comprehensive, open corpus of 601,257 privacy- and security-related app reviews, derived from 18.63 million raw reviews.
TaxoScale Pipeline: A novel methodology that combines recursive hierarchical clustering with LLM-based naming to scale taxonomy construction to hundreds of thousands of documents, overcoming the limitations of context windows and computational complexity.
Novel Discovery: The system successfully identifies branches absent from prior taxonomies, including a new Network Security branch (e.g., IP Address Management, VPN Functionality) and refined Behavioral Tracking subcategories (e.g., Driving Behavior, Financial Transaction Tracking).
Resource Release: The authors release the filtered corpus, extracted concern labels, and the TaxoScale codebase.

Results

TaxoScale was evaluated against four strong baselines: Chain-of-Layers, TaxoCom, TaxoAdapt, and SCYCHIC. The evaluation utilized four metrics:

Path Score: Measures if child nodes are genuine subtopics of parents.
Level Score: Measures the coherence and specificity consistency of siblings.
Coverage Score: The fraction of pseudo-labels that map to leaf nodes with high similarity.
Novelty Score: The extent to which nodes represent concepts beyond standard taxonomies.

Performance:

TaxoScale achieved the highest scores across all metrics: Path (0.7285), Level (0.7726), Coverage (0.9852), and Novelty (0.4778).
It notably outperformed the next-best baseline (TaxoAdapt) in Path and Level scores, and significantly exceeded Chain-of-Layers in Coverage (0.9852 vs. 0.9090).
The high novelty score indicates the system's ability to surface concepts not present in existing hand-crafted taxonomies.

Significance and Claims

The paper claims that TaxoScale addresses the critical gap between the massive scale of user-generated privacy data and the static nature of current taxonomies. By automating the expansion of expert-defined taxonomies, the system can keep pace with evolving threats and user perspectives.

The authors emphasize that their approach is domain-agnostic; while motivated by privacy and security, the pipeline makes no specific assumptions about these domains and can be applied to any large-scale corpus. However, they explicitly state that cross-domain validation is left for future work, as the current study is grounded in the specific challenges of mobile app privacy and security. The work demonstrates that combining recursive clustering with LLMs allows for the discovery of fine-grained, niche concerns (like "Driving Behavior Tracking") that would otherwise be absorbed into broader, less informative categories.