💻 computer science

Privacy-Preserving in Connected and Autonomous Vehicles Through Vision to Text Transformation

This paper proposes a novel privacy-preserving framework for Connected and Autonomous Vehicles that utilizes hierarchical reinforcement learning and vision-language models to transform sensitive visual data into refined textual descriptions, thereby protecting individual privacy while maintaining scene semantics and outperforming existing methods in both semantic accuracy and privacy metrics.

Original authors: Abdolazim Rezaei, Mehdi Sookhak, Ahmad Patooghy, Shahab S. Band, Amir Mosavi

Published 2026-01-15

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Abdolazim Rezaei, Mehdi Sookhak, Ahmad Patooghy, Shahab S. Band, Amir Mosavi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a world where smart cameras on street corners are constantly watching cars to catch traffic violators, like people speeding or not wearing seatbelts. These cameras are like super-observant security guards that can see everything inside a vehicle. While this helps keep roads safe, it creates a big problem: these guards are also peeking into people's living rooms (their cars), potentially stealing their identities or spying on their private lives.

This paper proposes a clever new way to solve this problem. Instead of sending the actual photos to a central computer, the system turns the photos into descriptive stories.

Here is how the paper's solution works, broken down into simple concepts:

1. The Problem: The "Blurry" Mistake

Usually, when we want to protect privacy in photos, we try to blur or pixelate faces. The paper argues this is like trying to hide a secret by scribbling over a map with a marker. It's messy, often inaccurate, and smart hackers can sometimes use AI to "erase" the scribbles and see the original face again.

2. The Solution: The "Translator" System

The authors propose a system that acts like a highly skilled translator.

The Input: A photo of a car interior.
The Output: A text description like, "A red sedan is stopped at a light; the driver is wearing a blue shirt and holding a phone."
The Magic: The system keeps all the useful traffic information (the car, the driver's actions) but completely removes the ability to recognize who the person is. It's like describing a person's outfit and actions without ever mentioning their name or showing their face.

3. How It Learns: The "Coach and Player" Game

The system doesn't just translate once and hope for the best. It uses a technique called Reinforcement Learning (RL), which is like a video game player learning from a coach.

The Player (The AI): Tries to write a description of the image.
The Coach (The Feedback Loop): Checks the description. If the description is too vague, the coach says, "Be more specific about the car." If the description accidentally reveals a face, the coach says, "Too much detail! Remove that."
The Iteration: The player tries again, gets feedback, and tries again. This happens in a loop, getting smarter and more precise with every round.

4. The "Library" Helper (RAG)

To make sure the descriptions are accurate and safe, the system uses a helper called RAG (Retrieval Augmented Generation). Think of this as a librarian who checks the player's story against a massive library of rules and examples.

If the player writes something that might accidentally reveal a secret, the librarian stops them and suggests a better, safer way to phrase it.
This ensures the final story is rich in detail but safe for public viewing.

5. The Results: Better Stories, Safer Privacy

The paper tested this system on two different sets of data (like two different groups of test subjects).

Privacy: When they tried to rebuild the original photos from the text descriptions, the results were terrible (in a good way). The reconstructed images looked nothing like the original people or cars. The "privacy score" was much higher than other methods.
Quality: Despite hiding the identities, the text descriptions were actually longer, richer, and more detailed than what other systems produced. They captured the scene perfectly without capturing the person.

Summary

Think of this technology as a privacy filter that turns a photograph into a novel. Instead of sending a risky photo to the cloud, the car sends a safe, detailed text story. The system uses a "coach" to refine the story and a "librarian" to double-check that no secrets are leaked. The result is a system that keeps traffic safe and data useful, while ensuring that what happens inside a car stays inside the car.

Technical Summary: Privacy-Preserving Vision-to-Text Transformation in Connected and Autonomous Vehicles

Problem Statement
Intelligent Transportation Systems (ITS) increasingly rely on Artificial Intelligence Equipped (AIE) cameras to monitor traffic, detect violations, and assess safety in Connected and Autonomous Vehicles (CAVs). While these systems enhance public safety, they capture imagery of vehicle interiors—a space widely regarded as private. Traditional privacy-preserving techniques, such as blurring, fogging, or face masking, are often insufficient. They can be inaccurate, fail to fully conceal sensitive regions, and are increasingly vulnerable to AI-based reconstruction attacks that can reverse obfuscation to reveal original identities. Furthermore, existing methods often operate directly in the image space, leaving residual visual data that can be exploited for identity theft, profiling, or unauthorized commercial use. There is a critical need for a robust solution that transforms sensitive visual data into a non-visual format without compromising the semantic utility required for traffic analysis.

Methodology
The paper proposes a novel framework that transforms visual data into descriptive text using a combination of Vision-Language Models (VLMs), hierarchical Reinforcement Learning (RL), and Retrieval Augmented Generation (RAG). Instead of processing images directly for classification or detection, the system converts them into privacy-aware textual descriptions. The methodology operates through four iterative steps:

VLM for Text Generation: A VLM is fine-tuned with privacy-sensitive supervision to convert images into initial textual descriptions. The model is trained to avoid generating personally identifiable information (PII) descriptors. It utilizes a Transformer-based architecture with self-attention mechanisms to encode visual inputs into token embeddings, guided by a loss function that penalizes the exposure of private details.
Hierarchical Prompt Selection via RL: To refine the initial text, a Reinforcement Learning agent (specifically using Proximal Policy Optimization, PPO) iteratively selects prompts from a hierarchical list. The agent operates in a state space defined by the semantic embedding of the current text and a privacy score. The action space involves selecting specific prompts to guide the VLM toward more precise and privacy-compliant descriptions. The reward function balances semantic similarity (using SBERT) against visual privacy (measured by reconstruction similarity via a Variational Autoencoder).
Feedback-Based Reward Mechanism (RAG): A Retrieval Augmented Generation (RAG) module acts as an external feedback loop. It retrieves relevant contextual information to validate the RL agent's output. This feedback is integrated into the RL objective function, allowing the model to learn from external knowledge signals rather than relying solely on internal rewards. This step corrects potential errors and ensures the generated text aligns with privacy objectives.
Iterative Refinement: The process repeats for a set number of iterations (e.g., three), progressively filtering redundant content and enhancing semantic density. The final output is a textual description that captures essential scene details (e.g., vehicle types, traffic conditions) while omitting sensitive visual cues.

Key Contributions
The authors identify three primary contributions:

Iterative Image-to-Text Architecture: The development of a framework that generates semantically rich, privacy-aware descriptive text from visual data, avoiding direct image exposure.
Hierarchical RL Mechanism: The design of a multi-stage RL loop that refines both prompts and generated outputs, improving precision beyond single-pass generation.
RAG-Integrated Feedback Learning: The integration of a feedback-based learning loop using RAG to validate and optimize prompt selection, enhancing the relevance and accuracy of the text while ensuring privacy preservation.

Results and Evaluation
The proposed method was evaluated on two datasets: CFP-FP (celebrity face verification) and AgeDB-30 (face verification with large age gaps). The system was compared against baseline models, including standard VLMs, PPO-based RL, and existing privacy-preserving methods (AdvFace, FLDATN).

Privacy Preservation: The proposed "Feedback-based RL" model demonstrated superior privacy protection. On the CFP-FP dataset, it achieved a Structural Similarity Index Measure (SSIM) of 0.85 and a Peak Signal-to-Noise Ratio (PSNR) of 22.18, outperforming other methods in minimizing visual similarity to the original image. The Mean Squared Error (MSE) was higher (365.2), indicating greater divergence from the original image, which is desirable for privacy. The Semantic Reconstruction Risk Assessment (SRRA) was lowest at 11.25%, suggesting a lower risk of identity recovery.
Text Quality and Semantic Richness: Despite the privacy focus, the model maintained high semantic utility. It generated significantly longer and more detailed descriptions than baselines, with a higher unique word count and a semantic similarity score of 0.7371 relative to the baseline, indicating strong alignment with the scene's meaning.
Differential Privacy: The study incorporated Differential Privacy (DP) by injecting Gaussian noise into sentence embeddings. Results showed that adding small noise levels (e.g., $\sigma=0.3$ ) significantly reduced the privacy budget ( $\epsilon$ ) while maintaining acceptable text quality (BLEU score of 0.756), establishing a measurable privacy-utility trade-off.
Ablation Study: Experiments confirmed that the full hierarchical feedback framework outperformed variants using only VLM, single-stage RL, or RL without RAG feedback, validating the necessity of both iterative refinement and external knowledge integration.

Significance and Claims
The paper claims to be the first attempt to combine Reinforcement Learning and Vision-Language Models specifically for transforming visual data into text as a means of privacy preservation in ITS. The authors argue that their approach offers a more robust solution than traditional blurring or obfuscation techniques, which are vulnerable to reconstruction attacks. By shifting the data representation from image space to semantic/textual space, the framework effectively prevents the exposure of sensitive visual details (such as faces or license plates) while retaining the operational information necessary for traffic monitoring. The study concludes that this method provides a scalable, adaptive mechanism for privacy-preserving visual-to-text transformation, suitable for deployment in smart city infrastructures and industrial automation environments. Future work is suggested to explore dynamic prompt generation for further personalization and generalizability.