The impact of abstract and object tags on image privacy classification

This paper investigates the effectiveness of abstract versus object tags for image privacy classification, revealing that abstract tags outperform object tags when tag budgets are limited, while both become equally useful when a larger number of tags are available.

The Gist

Imagine you are trying to explain a photograph to a friend over the phone so they can guess if the photo is private (like a diary entry) or public (like a postcard).

To do this, you have a limited "word budget." You can only use a few words to describe the picture. The big question this paper asks is: What kind of words work best?

Do you use Concrete Words (names of specific things, like "a red car," "a passport," or "a dog")?
Or do you use Abstract Words (feelings, vibes, and concepts, like "romance," "danger," "freedom," or "intimacy")?

Here is the breakdown of the research, explained simply.

The Two Types of Descriptions

The researchers looked at how computers "see" photos.

• Concrete Tags (The "What"): These are like a grocery list. They tell you exactly what objects are in the room. Example: "Man," "Sofa," "Passport."
• Abstract Tags (The "Vibe"): These are like a mood board. They tell you the feeling or the story. Example: "Family," "Celebration," "Secret," "Justice."

The Experiment: The "Word Budget" Game

The researchers tested these descriptions on three different types of photo datasets to see which words helped a computer decide if a photo was private. They played a game with a strict rule: You can only use a certain number of words.

Scenario A: You have a very small budget (1–5 words)

Imagine you can only say one or two words to describe a photo.

• The Result: If the photo is about feelings or personal secrets (like a couple kissing or a medical document), Abstract Words win.
• The Analogy: If you see a photo of a wedding ring and a blurry background, saying "Love" or "Commitment" tells you more about the privacy risk than saying "Metal" or "Finger." Abstract words capture the context and the story immediately.
• The Catch: If the photo is purely about spotting a specific object (like "Is there a gun in the room?"), Concrete words are still slightly better, but Abstract words are still surprisingly good.

Scenario B: You have a huge budget (13+ words)

Imagine you can write a whole paragraph describing the photo.

• The Result: It doesn't matter much which words you use. Concrete and Abstract words perform equally well.
• The Analogy: If you have enough words to say "A man holding a passport in a dark room looking nervous," you don't need the word "Fear" to know it's a private situation. The list of objects (Man, Passport, Room) is so long and detailed that it paints the full picture on its own. The "vibe" words become redundant because the "stuff" words have already told the whole story.

The Surprising Twist: Co-Occurrence

The researchers also looked at whether Abstract and Concrete words usually appear together.

• The Finding: They rarely appear together directly. For example, the word "Romance" (Abstract) doesn't always show up next to "Flowers" (Concrete) in the computer's data.
• The Lesson: This means Abstract words aren't just "copycats" of Concrete words. They are providing unique information that you can't get just by listing objects—but only if you don't have enough words to list everything.

The Big Takeaway for the Future

This research gives us a rule of thumb for building AI that protects our privacy:

1. If you are limited on space (like a quick alert or a mobile app): Don't just look for objects. You must teach the AI to understand the "vibe" (Abstract tags). It's the most efficient way to spot privacy risks when you can't say much.
2. If you have plenty of space (like a detailed report): You can stick to listing objects. It's easier for computers to find "Passports" and "Cars" than "Justice" or "Intimacy," and if you list enough of them, you get the same result.
3. The Sweet Spot: For the most accurate privacy protection, especially for subjective things (like "is this photo embarrassing?"), a mix of both is best. But if you have to choose, start with the Abstract words.

In short: If you have a short story to tell, use the feeling words. If you have a long story to tell, the object words will do just fine.

Technical Summary

1. Problem Statement

Image privacy classification is an inherently subjective and context-dependent task. While deep learning models often extract textual "deep tags" to describe image content for interpretability, most existing privacy classifiers rely heavily on concrete (object) tags (e.g., "passport," "car"). However, human decision-making regarding privacy involves both concrete entities and abstract concepts (e.g., "intimacy," "responsibility," "spirituality").

The paper addresses a critical gap in research: Which type of tag (concrete vs. abstract) is more effective for image privacy classification, and how does the quantity of tags available per image influence this effectiveness? Previous studies were limited by the use of user-generated tags (which are scarce) or lacked controlled comparisons regarding tag quantity and dictionary size.

2. Methodology

Datasets

The authors evaluated their approach on three distinct public datasets with different annotation strategies:

• PrivacyAlert: 6,800 images with binary labels (private/public) derived from subjective human judgments.
• VISPR: 22,112 images annotated with 67 attributes. Privacy is determined by the presence of specific object-guided attributes (e.g., documents, vehicles).
• DIPA2: 1,304 images with subjective and object-guided annotations. Annotators rated privacy threats for specific objects, which were aggregated into image-level labels.

Tag Extraction and Concreteness Definition

• Extraction: The authors used the commercial ClarifAI classifier to generate up to 200 tags per image from a fixed dictionary of 6,568 words.
• Categorization: Tags were classified as Abstract or Concrete based on the concreteness ranking by Brysbaert et al. [12] (scale 1–5).
  • Concrete ($B$): Score $\ge$ 4.75 (e.g., "skin," "machine gun").
  • Abstract ($A$): Score < 4.75 (e.g., "spirituality," "love").
  • Combined ($T$): Union of both sets.

Feature Selection and Fair Comparison

To ensure a fair comparison between tag types, the authors implemented a two-step feature selection process to control for dictionary size and tag sparsity:

1. Discriminative Selection: Calculated the $\chi^2$ score between tag probabilities and privacy labels. The top 1,000 ($M$) most discriminative tags were selected for each set ($A, B, T$) to equalize dictionary sizes.
2. Tag Sparsity Control: For a specific number of tags $k$ (ranging from 1 to 25), only the top-$k$ tags with the highest probabilities were retained for each image. This ensured that the only variable influencing performance was the abstractness of the tags, not the quantity of information.

Experimental Setup

• Model: A simple two-layer Multi-Layer Perceptron (MLP) with hidden layers of 128 and 32 neurons and ReLU activation.
• Variables: The study varied the number of tags ($k \in [1, 25]$) and the tag type ($A, B, T$).
• Metric: F1-macro score, averaged over 10 random seeds.

3. Key Results

Impact of Tag Quantity and Type

• Low Tag Budget ($k \le 10$):

  • Subjective Tasks (PrivacyAlert): Abstract tags significantly outperformed concrete tags. With $k \le 5$, abstract tags achieved an average 4.09 percentage point (p.p.) higher F1-score than concrete tags. A classifier using 5 abstract tags performed as well as one using 11 concrete tags.
  • Object-Guided Tasks (VISPR): Concrete and abstract tags performed similarly from the start, with abstract tags slightly underperforming (0.62–0.75 p.p.) at very low $k$. The task was generally "easier," showing high performance even with a single tag.
  • Hybrid Tasks (DIPA2): For small $k$, the combined set (abstract + concrete) yielded the best performance, suggesting a synergy between object detection and subjective interpretation.

• High Tag Budget ($k \ge 13$):

  • Across all datasets, the performance gap between abstract, concrete, and combined tags diminished significantly.
  • When a sufficient number of tags is available, concrete tags alone can achieve performance comparable to abstract tags.

Co-occurrence Analysis

The authors analyzed the Jaccard index between abstract and concrete tags to see if they naturally co-occur.

• Finding: Strong direct co-occurrence is rare (only ~100 pairs in PrivacyAlert/VISPR out of 1,000 possible pairs).
• Implication: The convergence in performance at high $k$ is not due to abstract and concrete tags simply describing the same objects simultaneously. Instead, as the number of tags increases, both types begin to convey similar informational value regarding privacy, likely by covering different aspects of the same context.

4. Key Contributions

1. Controlled Comparison: Provided the first rigorous, controlled comparison of abstract vs. concrete tags for privacy classification, eliminating biases related to dictionary size and tag count.
2. Contextual Insight: Demonstrated that the optimal tag type depends on the subjectivity of the task and the available tag budget.
   • For subjective tasks with limited tags, abstract concepts are superior.
   • For object-guided tasks or when many tags are available, concrete objects are sufficient.
3. Scalability: Moved beyond user-generated tags (which are scarce) to automatically generated deep tags, making the findings applicable to large-scale automated systems.

5. Significance and Future Implications

• Efficiency in Privacy Classifiers: For applications where computational resources or tag generation budgets are limited (e.g., real-time mobile privacy filters), prioritizing abstract tags is crucial for subjective privacy tasks.
• Interpretability: The findings suggest that interpretable AI systems explaining privacy decisions should not rely solely on object detection. Including abstract concepts (e.g., "intimacy," "vulnerability") provides richer context, especially when only a few explanatory tags can be shown to a user.
• Design Guidelines:
  • Small Tag Budget: Use abstract tags for subjective tasks; use combined tags for hybrid tasks.
  • Large Tag Budget: Concrete tags are a viable, cost-effective alternative to abstract tags without significant performance loss.
• Future Research: Encourages the development of classifiers that dynamically select tag types based on the task's subjectivity and the available computational budget.