Technical Summary: Automated Generation of Accurate Privacy Captions from Android Source Code Using Large Language Models

Problem Statement

App privacy notices are critical artifacts that inform users and regulators about what personal information an app collects, how it is used, and why. These notices often rely on "privacy captions"—succinct natural language sentences describing these behaviors. However, creating accurate captions is a significant burden for developers, particularly those in small teams without privacy expertise. Current practices, such as using online policy generators, copying from similar apps, or manually annotating code, frequently result in captions that are inaccurate, incomplete, or inconsistent with the app's actual behavior. This discrepancy can lead to regulatory fines and app removal.

Existing automated approaches to generating privacy notices suffer from three primary limitations:

1. High Developer Burden: Many tools require extensive manual input, such as annotating source code or answering detailed questionnaires.
2. Limited or Inaccurate Context: Automated methods often rely on restricted source code contexts (e.g., only the first few methods in a call graph) or potentially inaccurate inputs like privacy policies, which are known to often misrepresent actual app behavior.
3. Format Specificity: Many approaches generate specific notice formats (e.g., legalistic privacy policies) that are difficult to adapt into other formats like permission rationales or app store descriptions.

Methodology: PCapGen

The authors propose PCapGen (Privacy Caption Generator), a framework designed to automatically identify privacy behaviors in Android source code and generate accurate, concise, and complete privacy captions without requiring developer input. The framework consists of three main components:

1. Identifier: Heuristics-Based Taint Analysis

The Identifier component aims to trace the flow of personal information (taint paths) within an app.

• Challenge: Traditional taint analysis tools (like FlowDroid) rely on pre-defined lists of "source" (data entry) and "sink" (data exit) methods. They struggle to identify paths involving unknown sink methods, which are common in custom app logic.
• Innovation: The authors developed a novel heuristics-based approach. Instead of relying solely on pre-listed sinks, they construct a call graph from a known source API up to a fixed depth (determined to be six levels based on empirical analysis of 50 apps). All methods within this depth are treated as potential sink candidates. Taint analysis is then run to extract valid paths.
• Scope: This approach was tested on 80 unlisted non-location API sources (covering WiFi, account, network, camera, storage, and sensors). It successfully identified valid taint paths for WiFi, network, and account APIs, though it faced limitations with camera and storage APIs due to asynchronous callbacks or component-based flows not captured in the call graph.

2. Extractor: Source Code Context Construction

The Extractor converts the identified taint paths into a human-readable source code context suitable for Large Language Models (LLMs).

• Decompilation: Using the reverse engineering tool Androguard, the system decompiles APK files to extract the source code of classes and methods involved in the taint paths.
• Statement Mapping: Taint paths are initially represented in Jimple (an intermediate, non-human-readable format). The Extractor uses an LLM (specifically DeepSeek in the final implementation) with In-Context Learning (ICL) to map these Jimple statements back to the corresponding lines in the extracted source code.
• Output: This process yields a "coarse-grained" view (the full class/method structure) and a "fine-grained" view (specific statements handling personal data), providing the LLM with a comprehensive context spanning multiple files and methods.

3. Generator: Caption Generation

The Generator uses an LLM to synthesize the extracted context into a privacy caption.

• Prompt Engineering: The authors iterated through various prompting strategies. They initially attempted "prompt chaining" (breaking the task into sub-steps: identifying statements, creating labels, then generating captions) but found it led to error propagation and hallucinations.
• Final Approach: They adopted a single-prompt approach using ICL. The prompt provides the model with the source code, the taint analysis statements, and an example tuple of code and a caption. The model is then tasked with generating a caption for the new code sample.
• Configurations: The system was evaluated using three different LLMs: GPT-4, Claude Opus 4, and DeepSeek.

Dataset and Evaluation

To evaluate PCapGen, the authors constructed a dataset of 442 unique Android source code samples with developer-verified baseline captions.

• Baseline Creation: Since manually writing captions for hundreds of code samples is infeasible, they used GPT-4 to generate initial baseline captions.
• Verification: Professional Java developers (screened via Prolific) verified these captions. They rated them on accuracy, conciseness, and completeness, and selected the preferred version between two generated options. Only samples where captions met a quality threshold were retained.
• Evaluation Metrics: The performance of PCapGen was compared against these baseline captions using:
  • LLMs-as-Judge: Three models (LlaMa 3.3, Mistral, Gemini 2.5 Pro) rated captions on the three quality criteria and chose the better caption in pairwise comparisons.
  • Human Privacy Experts: Two privacy experts performed similar ratings and preference selections.
  • Automated Metrics: BERTScore, BARTScore, and ROUGE-L were used to measure semantic and token-level similarity.

Key Results

1. Configuration Comparison (RQ1)

Among the three LLM configurations, PCapGen_Claude (using Claude Opus 4) consistently outperformed PCapGen_GPT and PCapGen_DeepSeek across all quality criteria (accuracy, conciseness, completeness) according to all LLM judges. Statistical analysis (paired t-tests) confirmed that the difference between PCapGen_Claude and the other configurations was statistically significant with a medium effect size.

2. Semantic Similarity (RQ2)

PCapGen captions were found to be semantically very similar to the developer-curated baseline captions.

• Human/LLM Ratings: Privacy experts and LLM judges rated the similarity as moderate to high (average scores > 3.3 on a 1-5 scale).
• Automated Metrics: High BERTScore and BARTScore indicated strong semantic overlap. However, ROUGE-L scores were low (precision ~0.33), indicating that while the meaning was similar, the vocabulary and phrasing differed significantly. This suggests PCapGen successfully rephrases behaviors into more concise and natural language.

3. Quality Comparison with Judges (RQ3)

LLM judges consistently rated PCapGen_Claude captions higher than baseline captions on all three criteria:

• Accuracy: +0.57 improvement.
• Completeness: +0.47 improvement.
• Conciseness: +0.35 improvement.
  These improvements were statistically significant (p < 0.001) with large effect sizes. In pairwise preference tests, LLM judges preferred PCapGen_Claude over the baseline in 76% to 100% of cases.

4. Quality Comparison with Experts (RQ4)

Privacy experts did not find a statistically significant difference in the mean ratings of the three quality criteria between PCapGen_Claude and the baseline. However, in pairwise preference tasks, experts chose PCapGen_Claude over the baseline 71% of the time. This suggests that while experts may rate the two similarly on a scale, they perceive the PCapGen captions as superior when directly comparing them.

Significance and Contributions

The paper claims the following primary contributions:

1. Novel Framework: PCapGen is the first approach to complement traditional static/taint analysis with LLMs to automatically generate privacy captions directly from source code without developer input.
2. Extended Taint Analysis: The authors developed a heuristics-based method to identify taint paths between known sources and unknown sinks, expanding the capabilities of traditional tools that rely on pre-defined lists.
3. Robust Evaluation: The study provides a comprehensive evaluation framework using both human privacy experts and LLMs-as-judge, demonstrating that PCapGen generates captions that are as good as or better than those curated by developers.
4. Dataset and Tools: The authors released a high-quality dataset of 442 Android code snippets with privacy captions, along with the scripts and annotation tools used in the study.

Conclusion

The authors conclude that PCapGen demonstrates the feasibility of automatically generating high-quality privacy captions from Android source code. By leveraging a large source code context derived from taint analysis and the reasoning capabilities of LLMs, the system produces captions that are concise, complete, and accurate. The results suggest that this approach can significantly aid resource-constrained developers in fulfilling regulatory obligations by automating the creation of privacy notices that accurately reflect app behavior.