Bodhi VLM: Privacy-Alignment Modeling for Hierarchical Visual Representations in Vision Backbones and VLM Encoders via Bottom-Up and Top-Down Feature Search

1. Problem Statement

Privacy-preserving learning systems often inject noise into hierarchical visual representations (e.g., feature pyramids in object detectors or layers in Vision-Language Model (VLM) encoders) to protect sensitive data. However, a central challenge remains: how to interpret and model the alignment between observed layer-wise perturbations and a declared privacy budget ($\epsilon$).

Existing methods often focus on training with Differential Privacy (DP) or post-hoc auditing of model outputs. There is a lack of frameworks that:

• Operate at the feature level within hierarchical neural structures.
• Provide an interpretable signal regarding how well the injected noise aligns with a specific reference mechanism (e.g., Laplace or Gaussian noise scaled by $c/\epsilon$).
• Function as a modeling tool for representation learning rather than just a binary privacy certification.

Note: The authors explicitly state that Bodhi VLM does not provide formal end-to-end Differential Privacy certification or estimate the "effective $\epsilon$" of a deployed mechanism. Instead, it offers a reference-relative discrepancy score to assess alignment.

2. Methodology: The Bodhi VLM Framework

The framework consists of three core components designed to process hierarchical features from vision backbones (e.g., YOLO, DETR) and VLM vision towers (e.g., CLIP, LLaVA, BLIP).

A. Sensitive Feature Localization (BUA & TDA)

The system identifies which features correspond to sensitive concepts (e.g., "person," "license plate") using two traversal strategies over multi-scale feature maps:

1. Bottom-Up Analysis (BUA): Starts from low-level layers (e.g., patch embeddings) and aggregates upward.
2. Top-Down Analysis (TDA): Starts from high-level semantic layers and propagates downward, linking groups across layers via set intersections.

Clustering Mechanism:
Both strategies utilize:

• Sensitivity Scoring: Features are scored based on their relevance to a sensitive label set $S_\epsilon$ (e.g., max detector confidence or cosine similarity to text prototypes).
• MDAV (Maximum Distance to Average Vector): Used for microaggregation to form stable groups and enforce minimum group sizes.
• NCP (Normalized Certainty Penalty): A quantitative metric used to measure within-group generalization and ensure the grouping is stable.
• Output: At each layer $i$, features are partitioned into a Sensitive Group ($G^{(s)}_i$) and a Non-Sensitive Group ($G^{(n)}_i$).

B. Expectation-Maximization Privacy Assessment (EMPA)

Once sensitive groups are identified, EMPA evaluates the alignment of the observed noise with a declared budget.

• Modeling: It fits a K-component diagonal-Gaussian mixture model to the observed sensitive feature vectors.
• Reference Comparison: It compares the fitted parameters ($\Theta_\epsilon$) against a reference distribution ($\Theta^{ref}_\epsilon$) generated by the evaluator's specified mechanism (e.g., Laplace noise with scale $c/\epsilon$).
• Output Metrics:
  • $BAS_\epsilon$: A reference-perturbation discrepancy score (L2 distance between mixture parameters), indicating how well the observed noise matches the reference.
  • $Bias_\epsilon$: A weight-only diagnostic measuring the deviation of mixture weights from a uniform distribution.

3. Key Contributions

1. Formalization of Hierarchical Sensitive Search: The paper formalizes BUA and TDA strategies for searching sensitive features across Feature Pyramid Networks (FPN) and VLM vision backbones using NCP and MDAV clustering.
2. EMPA Module: Introduction of a feature-level budget-alignment estimator that provides an interpretable discrepancy signal, distinct from formal DP certification.
3. Cross-Architecture Applicability: The framework is validated without modifying the underlying model architecture, applying successfully to diverse backbones (YOLO, DETR, PPDPTS) and VLM vision towers (CLIP, LLaVA, BLIP).
4. Comprehensive Evaluation: The authors establish a rigorous experimental protocol comparing Bodhi VLM against generic distributional baselines (Chi-square, K-L divergence, MMD) and task-specific baselines (MomentReg, NoiseMLE, Wass-1).

4. Experimental Results

Experiments were conducted on MOT20 and COCO datasets using various privacy budgets ($\epsilon \in \{0.1, 0.01, 0.001\}$).

• BUA vs. TDA: Both strategies yield comparable deviation trends. BUA is slightly more sensitive to noise levels, while TDA performs slightly better in specific alignment metrics on certain datasets.
• EMPA Performance:
  • EMPA provides a stable budget-alignment signal (low rMSE) across different detectors and VLMs.
  • In synthetic experiments, EMPA bias remained stable ($\approx 0.89$) across different $\epsilon$ values, indicating that the metric effectively captures structural discrepancies rather than just noise magnitude.
  • Structured grouping (BUA/TDA) produced significantly more interpretable signals than random partition baselines.
• Comparison with Baselines:
  • EMPA outperformed or matched generic metrics (Chi-square, K-L, MMD) in terms of stability and alignment error (rMSE) relative to the Laplace reference.
  • Unlike training-time DP methods (ViP, DP-Cap) or de-identification tools (VisShield), Bodhi VLM serves as a post-hoc audit tool for feature-level noise alignment.
• VLM Validation: The framework successfully assessed privacy alignment on CLIP, LLaVA, and BLIP vision encoders, demonstrating transferability across ViT and ResNet architectures.

5. Significance and Limitations

Significance:

• Interpretability: Moves beyond "black box" privacy auditing by visualizing and quantifying how noise distributes across specific layers and sensitive concepts.
• Flexibility: Does not require retraining models; it can be applied to any existing vision backbone or VLM encoder.
• New Paradigm: Shifts the focus from "does this model satisfy DP?" to "does the observed feature perturbation align with the claimed privacy mechanism?"

Limitations:

• Reference-Relative: The assessment is only valid relative to the evaluator's chosen reference mechanism ($M_{ref}$). If the actual deployed mechanism differs, the score reflects a mismatch.
• No Formal DP Guarantee: It does not certify end-to-end differential privacy.
• Scope: Currently validated only on vision encoders/towers. It does not yet assess full cross-modal fusion or language-decoding pathways in VLMs.
• Partition Dependence: In some synthetic settings, the discrepancy signal is driven more by the induced group structure than the perturbation scale itself.

Conclusion

Bodhi VLM introduces a novel, learnable, and interpretable framework for modeling privacy-aligned hierarchical representations. By combining bottom-up and top-down feature search with an Expectation-Maximization assessment, it provides a robust tool for auditing how well privacy noise aligns with declared budgets in modern vision and vision-language models.