PrivPRISM: Automatically Detecting Discrepancies Between Google Play Data Safety Declarations and Developer Privacy Policies

The paper introduces PrivPRISM, an automated framework that uses language models to detect widespread discrepancies between Google Play's simplified data safety declarations and developers' full privacy policies, revealing that over half of popular apps contain non-compliant or misleading disclosures about their data practices.

The Gist

Imagine you walk into a restaurant. On the menu, there's a tiny, easy-to-read sticker that says: "We use your name and your credit card to take your order." It's simple, clear, and fits on a sticky note.

But then, you look at the full legal contract (the Privacy Policy) hidden in the back of the menu. It's 50 pages long, written in tiny, confusing font. When you finally squint to read it, you realize it actually says: "We also track your GPS location to see where you live, we sell your browsing history to advertisers, and we keep a record of every time you visit the bathroom."

The sticker (the Data Safety Declaration) and the contract (the Privacy Policy) tell two completely different stories.

This is exactly the problem researchers at the University of Sydney and UNSW tackled in their paper, PrivPRISM.

The Problem: The "Menu" vs. The "Fine Print"

Google Play (the Android app store) realized that nobody reads those 50-page contracts. So, they forced developers to put up a simple "sticker" (Data Safety Declaration) summarizing what data they collect.

However, developers are sneaky. They often:

1. Lie on the sticker: They leave out scary stuff to make the app look safe.
2. Hide in the fine print: They put the scary stuff in the long contract, hoping you won't read it.
3. Copy-paste: Many developers use the exact same contract for 20 different games, even if the games do totally different things.

This creates a "Trust Gap." You think you're safe because of the sticker, but the contract says otherwise.

The Solution: PrivPRISM (The "Super-Translator")

The researchers built a robot brain called PrivPRISM. Think of it as a super-intelligent detective that speaks two languages: "Legal-ese" (the long contracts) and "Sticker-ese" (the short summaries).

Here's how it works, using a simple analogy:

1. The Decoder (The Translator): Imagine a translator who reads the 50-page contract and breaks it down into bite-sized sentences. "Okay, this paragraph says they collect your location."
2. The Encoder (The Sorter): Imagine a sorter that takes those sentences and tags them. "This is about Location. This is for Advertising."
3. The Verifier (The Fact-Checker): This is the clever part. Sometimes the translator gets confused or makes things up (hallucinations). The Verifier is a second robot that double-checks the work to make sure the translator didn't lie.

PrivPRISM then takes the "Sticker" (what the developer said they do) and compares it side-by-side with the "Contract" (what the robot found they actually do).

What Did They Find? (The Shocking Results)

The team used PrivPRISM to check 7,770 popular mobile games and 1,700 other apps. The results were eye-opening:

• The "Lie" Rate: In 53% of the games, the sticker and the contract didn't match. In other types of apps, it was even worse (61%).
• The "Missing" Secrets: The contracts often admitted to collecting sensitive things like financial info and exact location, but the stickers said, "Oh, we don't touch that!"
• The "Copy-Paste" Scam: They found that 65% of apps were using the exact same privacy policy as other apps. It's like a pizza place, a car dealership, and a dentist all using the exact same menu, even though they sell totally different things.
• The "Ghost" Links: In many cases, the link to the privacy policy was broken, or it led to a page that just said "Hello World" (a placeholder), or it redirected you through 5 different websites before you could even see the policy.

A Real-World Example

The paper highlights a game called "Farm Studio" with over 100 million downloads.

• The Sticker: Says they collect your name and device ID.
• The Contract: Says they collect your name, but then claims, "Don't worry, this data never leaves your phone!"
• The Reality: The robot found that the app actually collects financial data and app performance data, which wasn't on the sticker at all.
• The Developer: The developer's website was just a blank page. They were essentially hiding behind a mask.

Why Should You Care?

You might think, "I don't read the contracts anyway, so what's the big deal?"

The problem is that you are giving "blanket consent." When you click "I Agree" on a vague sticker, you are legally agreeing to whatever is in the hidden contract. If the contract says they can sell your location to the highest bidder, and the sticker didn't mention it, you are still legally bound by that contract.

The Takeaway

PrivPRISM is a tool that shines a light into the dark corners of app stores. It proves that:

1. Developers are often not telling the truth (or at least, not the whole truth).
2. The current system relies too much on trust, and that trust is being abused.
3. We need automated police (like PrivPRISM) to check the work, because humans can't read millions of contracts.

The Bottom Line: The next time you download a game, remember: The little "Data Safety" sticker is just the tip of the iceberg. The real story is usually hidden in the deep, dark water below, and unless we have tools like PrivPRISM to dive down there, we'll never know what's really happening to our data.

Technical Summary

Here is a detailed technical summary of the paper "PrivPRISM: Automatically Detecting Discrepancies Between Google Play Data Safety Declarations and Developer Privacy Policies."

1. Problem Statement

Mobile app developers are mandated by platforms like Google Play to provide simplified "Data Safety" (DS) declarations alongside full "Privacy Policies" (PP). While DS labels are designed to be user-friendly summaries, they are self-declared and often contradict the detailed legal text found in PPs.

• The Core Issue: There is a lack of automated, scalable tools to verify the consistency between these two self-attested disclosures. Regulatory frameworks (GDPR, CCPA) require consistency, yet manual audits are infeasible at scale.
• Limitations of Existing Work: Previous methods rely solely on either embedding-based models (noisy semantic search) or autoregressive decoder models (lower precision in retrieval). Furthermore, existing research treats PP and DS analysis as separate branches, lacking a unified framework to measure cross-declaration consistency or compare them against actual app behavior (code).

2. Methodology: The PrivPRISM Framework

PrivPRISM (Privacy Policy Reasoning and Investigation using Systematic language-Modelling) is a hybrid framework combining encoder and decoder language models to extract, structure, and compare data practices.

A. Terminology & Definitions

The framework standardizes data extraction into three categories consistent with Google's definitions:

1. Data Practice: The activity (e.g., Collection, Sharing).
2. Data Item: Specific information (e.g., Location, Device ID, Financial Info).
3. Data Purpose: The reason for processing (e.g., Analytics, Advertising, App Functionality).

B. System Architecture

The pipeline consists of four main modules:

1. Heading and Paragraph Extraction:
   • Uses a generative LLM (Llama3.1-8B-Instruct) to identify section headings in raw HTML privacy policies.
   • Segments the policy into paragraphs, merging short paragraphs to ensure context coherence.
2. Explained Data Practice Classification (Encoder-Decoder Hybrid):
   • Encoder (PrivBERT): Fine-tuned on the OPP-115 dataset to classify paragraphs into high-level categories (e.g., First-party collection, Third-party sharing). This ensures high classification accuracy.
   • Decoder (Llama3.1): Generates textual explanations (excerpts from the policy) to justify the classification, adding interpretability.
3. Fine-Granular Data Practice Decoding:
   • Uses a decoder model to extract five specific elements from classified paragraphs: Data, Purpose, Processing, Retention, and Recipients.
   • This step transforms unstructured text into structured data tuples.
4. Keyword Mapping & Self-Supervised Verification:
   • Mapping: Maps extracted free-text data items/purposes to a fixed set of 23 keywords (Data Items) and 8 keywords (Purposes) defined by Google's DS schema.
   • Verification: To prevent "hallucinations" (where the decoder invents keywords), a lightweight encoder-based verifier is trained on the decoder's outputs. This verifier acts as a self-supervised filter to ensure one-to-one mapping accuracy and correct errors before final aggregation.

C. Compliance Metrics

PrivPRISM defines two primary metrics to quantify discrepancies:

• PP Compliance: The proportion of DS-declared data items that are explicitly found in the Privacy Policy.
• DS Compliance: The proportion of PP-mentioned data items that are reflected in the DS declaration.
• Purpose Compliance: Intersection-over-Union (IoU) of purposes associated with specific data items across both documents.

3. Key Contributions

1. Novel Framework: Introduction of PrivPRISM, the first systematic framework to jointly leverage encoder (for precision) and decoder (for explainability) models for fine-grained policy extraction and cross-verification.
2. Performance Benchmarking: PrivPRISM outperforms state-of-the-art baselines (including fine-tuned Llama3.1 and zero-shot GPT-4o), achieving 6 percentage points higher precision in data practice classification and reducing mapping errors by 22.3 percentage points.
3. Large-Scale Empirical Analysis: Application of the framework to 7,770 popular mobile games and 1,711 generic non-game apps on Google Play.
4. Multi-Source Validation: Integration of static code analysis (APK manifest and method calls) to compare declared policies against actual app permissions and data flows.

4. Key Results

The study analyzed ~10,000 apps (via ~5,000 unique privacy policies) and revealed systemic non-compliance:

• High Discrepancy Rates:
  • 53% of mobile games and 61% of non-game apps showed inconsistencies between their Privacy Policies and Data Safety declarations.
  • Only ~46.7% of game pairs achieved 100% alignment.
• Under-Declaration in DS Labels:
  • Developers consistently disclose more data in PPs than in DS labels.
  • Static Code Analysis: While PPs disclosed 66.8% of potential sensitive data accesses (e.g., location, financial info), DS declarations covered only 36.4%.
  • Code Evidence: Many apps access sensitive data (Financial: 84.3%, Location: 98.3%) but fail to declare these in DS labels, suggesting developers use vague PPs to secure blanket consent while under-reporting in DS.
• Policy Reuse and Genericization:
  • 64.9% of apps reuse generic privacy policies across multiple titles.
  • This leads to "catch-all" disclosures that obscure app-specific risks and inflate apparent discrepancies when analyzed per-app.
• Purpose Misalignment:
  • Purpose compliance is extremely low (almost no cell exceeded 50%).
  • Developers often fail to specify why data is collected in DS labels (e.g., location data for advertising is rarely explicitly linked in DS, despite being common in PPs).
• Manual Audit Findings (50 Lowest Compliance Games):
  • 38% of policy links required redirection or led to inactive pages.
  • 33% showed mismatches between the developer name and the policy owner.
  • Cases of "placeholder" policies (e.g., "Hello World" URLs) and contradictory statements (e.g., "We collect X" vs. "X never leaves the device").

5. Significance and Implications

• Regulatory Enforcement: The findings highlight that current self-declaration mechanisms are insufficient. Automated enforcement tools like PrivPRISM are necessary to detect non-compliance at scale, moving beyond reactive, complaint-based enforcement.
• User Trust: The widespread use of vague, contradictory, or generic policies undermines user trust and informed consent. Users are often presented with "blanket consenting" scenarios due to the lack of specific, aligned disclosures.
• Platform Integrity: The prevalence of under-declared sensitive data (especially financial and location) in high-profile apps (100M+ downloads) poses significant privacy risks.
• Future Directions: The paper suggests future work should include dynamic analysis (runtime data flows) to complement static analysis and extend the framework to cross-platform (iOS vs. Android) comparisons.

In conclusion, PrivPRISM demonstrates that a significant portion of the mobile app ecosystem is non-compliant with consistency regulations, driven by developer practices of policy reuse, vague disclosures, and systematic under-reporting in user-facing safety labels.