What Your Posts Reveal: A Benchmark and Agentic Framework for User-Level Privacy Leakage on Social Media

Technical Summary: SopriBench and Argus for User-Level Privacy Leakage

Problem Statement
Current research on social media privacy largely focuses on post-level self-disclosure or the detection of explicit Personally Identifiable Information (PII) within single posts. However, the paper argues that user-level privacy leakage is fundamentally cumulative and cross-post. Weak cues scattered across text, images, and metadata—harmless in isolation—can jointly expose sensitive user attributes such as home addresses, workplaces, daily routines, and social relationships when aggregated.

Existing approaches suffer from two primary gaps:

1. Lack of a Unified Benchmark: There is no standardized benchmark for user-level multimodal privacy leakage that accounts for the accumulation of weak cues across multiple posts and modalities. Real-user datasets are difficult to release due to sensitivity, and existing synthetic datasets often remain at the post level.
2. Inadequate Evaluation Metrics: Standard evaluations rely on binary accuracy (correct/incorrect), which fails to capture the severity of exposure. A coarse inference (e.g., city-level location) and a fine-grained inference (e.g., specific residential compound) are treated equally, despite vastly different privacy risks. Furthermore, sensitivity varies by context (e.g., inferring a minority sexual orientation is more sensitive than inferring a majority one).

Methodology

1. SopriBench: A Synthetic Benchmark
To address the data gap, the authors propose SopriBench, a controllable synthetic benchmark guided by leakage patterns abstracted from a private reference corpus of 161 Rednote and 157 Instagram accounts.

• Construction: The benchmark contains 50 synthetic user profiles, 500 multimodal posts, and 1,569 images. It avoids copying real user data; instead, it uses de-identified patterns (scenarios, clue carriers, leakage forms) to generate synthetic content.
• Annotations: Each post is annotated with ground-truth attributes, contextual sensitivity (1–5 scale), value granularity (e.g., country to address), leakage type (explicit, implicit, cross-post), inference difficulty, and supporting evidence.
• Privacy Exposure Score (PES): The authors introduce PES as a novel evaluation metric. Unlike binary accuracy, PES weights the granularity of the inferred value by its contextual sensitivity.
  $$PES = \frac{\sum_j g_j \cdot s_j}{\sum_j s_j}$$
  Where $g_j$ is the granularity score (0 to 1) and $s_j$ is the sensitivity score. This metric distinguishes between coarse and fine-grained exposure and prioritizes sensitive attributes.

2. Argus: An Agentic Inference Framework
The paper proposes Argus, a training-free agentic framework inspired by abductive reasoning (inference to the best explanation). Argus treats privacy inference as an iterative, evidence-grounded investigation rather than a one-shot classification task.

• Cross-Post Evidence Graph (CPeg): Argus maintains a persistent graph structure storing posts, raw evidence, hypotheses, and their citation/support relations. This allows the system to connect weak cues across multiple posts.
• Workflow:
  1. Perceiver: Skims public posts to extract raw evidence (captions, OCR candidates, visual tags) and initializes the CPeg.
  2. Hypothesizer: Generates candidate hypotheses for private attributes based on accumulated evidence. Hypotheses are stored as Candidate (unverified) or Unresolved (checked but insufficient).
  3. Investigator: Routes active hypotheses to appropriate tools (OCR, web search, map search, visual re-inspection) to collect missing evidence.
  4. Verifier: Validates collected evidence against the hypothesis. Only grounded evidence is admitted; unsupported hypotheses are suspended or removed.
  5. Profile Projection: Converts sufficiently supported hypotheses into derived evidence and projects them into a final privacy profile.
• Key Mechanism: The framework explicitly tracks uncertainty. It does not commit to a specific attribute value until evidence chains across posts and modalities converge to support it.

Results

Experiments were conducted on SopriBench comparing Argus against five baselines: TextLLM, PostVLM, SelfDisc, Holmes (visual profiling), and SingleAgent (a multimodal tool-using agent without the specific Argus architecture).

• Performance: Argus achieved a PES of 0.55, representing a 25% relative improvement over the strongest baseline, SingleAgent (0.44).
• Cross-Post Leakage: The most significant gains were observed in cross-post leakage scenarios, where Argus improved PES by +0.17 (from 0.39 to 0.56) compared to SingleAgent. This highlights the framework's ability to connect scattered cues.
• Ablation Studies:
  • Removing the Verifier increased binary accuracy (0.71 to 0.74) but lowered PES (0.55 to 0.50), confirming that binary accuracy rewards speculative, coarse guesses while PES favors specific, sensitive, and verified exposures.
  • Removing the CPeg caused a sharp drop in cross-post PES (0.56 to 0.38), validating the necessity of persistent evidence memory for cumulative leakage.
  • Removing the Investigator (tool routing) resulted in the largest overall performance drop, emphasizing the need for active evidence collection.

Significance and Claims

The paper claims to make three primary contributions:

1. Problem Identification: It establishes that user-level privacy risk on public social media arises from the cumulative nature of weak cues across posts and modalities, rather than explicit disclosure in single posts.
2. Benchmark and Metric: It introduces SopriBench and PES, providing the first controllable benchmark for user-level multimodal leakage and a metric that captures exposure severity beyond binary accuracy.
3. Agentic Framework: It presents Argus, a training-free framework that models privacy inference as an abductive reasoning process. By maintaining explicit hypotheses and verifying evidence through a graph structure, Argus effectively aggregates cross-post cues.

The authors conclude that evaluating public social media privacy requires shifting from isolated post-level disclosure detection to a user-level, multimodal, and cross-post inference paradigm. They position their work as a step toward better understanding and protecting user privacy in the era of cumulative data exposure, while noting limitations regarding the synthetic nature of the benchmark and the evolving capabilities of the underlying foundation models.