Missing-by-Design: Certifiable Modality Deletion for Revocable Multimodal Sentiment Analysis

The paper introduces Missing-by-Design (MBD), a unified framework for revocable multimodal sentiment analysis that combines structured representation learning with a certifiable parameter-modification pipeline to enable the machine-verifiable deletion of specific data modalities while maintaining predictive performance and privacy compliance.

The Gist

Imagine you have a very smart, all-knowing assistant who helps you analyze how people feel based on three things they say: what they write (text), how they sound (audio), and what their face looks like (video). This assistant is great at guessing emotions, but it has a problem: it memorizes everything, including private details you might not want it to keep forever.

What if you wanted to say, "Hey, please forget everything you learned from my voice, but keep the rest of your knowledge about my text and face"? Usually, to do this, you'd have to fire the assistant and hire a whole new team to train from scratch. That's expensive, slow, and wasteful.

This paper introduces a new system called Missing-by-Design (MBD). Think of it as a "surgical eraser" that can remove specific memories from your AI assistant without firing the whole team.

Here is how it works, broken down into simple concepts:

1. The "Universal Translator" vs. The "Specialist"

Most AI models mix all their information together in a big blender. If you want to remove the "audio" flavor, you accidentally ruin the "text" flavor too.

MBD's Secret Sauce: It separates the information into two buckets:

• The "Sample-Specific" Bucket: This holds the unique details of this specific sentence (e.g., "I am angry right now").
• The "Property" Bucket: This holds the general rules of how that type of data works (e.g., "How human voices generally express sadness").

By keeping these separate, the AI can say, "I know how to understand voices in general (Property), but I will forget the specific voice patterns from this user."

2. The "Magic Reconstructor"

What happens if the AI forgets the audio? Won't it be confused?
MBD has a built-in Magic Reconstructor. Imagine a painter who has seen thousands of faces. If you cover up the eyes in a photo, the painter can guess what the eyes probably looked like based on the rest of the face.

• If the audio is missing, the AI uses the text and video to "hallucinate" (reconstruct) what the audio would have sounded like.
• This ensures the AI stays smart and accurate even when parts of the data are missing or deleted.

3. The "Surgical Scalpel" (The Deletion)

When you ask to delete the audio, MBD doesn't just smash the audio part of the brain. It performs surgery:

1. Identify the Culprits: It looks at the neural network and finds the specific "wires" (parameters) that are most responsible for remembering audio.
2. The Cut: It carefully cuts or dampens those specific wires.
3. The Noise Injection: To make sure the AI truly "forgets" and doesn't just pretend to, it adds a tiny, controlled amount of static (noise) to those wires. This is like scrambling a specific page in a book so the story is gone, but the rest of the book remains readable.

4. The "Certificate of Deletion" (The Receipt)

This is the coolest part. In the real world, if you ask a company to delete your data, how do you know they actually did it? They could just say "Okay" and keep it.

MBD generates a Modality Deletion Certificate (MDC).

• Think of this as a digital receipt or a notarized deed.
• It is a machine-readable document that proves, mathematically, that the specific audio data has been removed.
• It lists exactly which wires were cut, how much noise was added, and provides a "mathematical guarantee" that the AI can no longer distinguish your specific audio from random noise. You can verify this certificate yourself!

Why is this a big deal?

• Privacy: You can now demand that an AI forgets your voice or face without losing its ability to understand your text.
• Efficiency: Instead of retraining the whole AI (which takes days and costs a fortune), this "surgery" takes seconds.
• Trust: The certificate gives you proof, not just a promise.

In a nutshell: MBD is like having a librarian who can take a specific book out of a library, shred the pages you don't want, and give you a notarized receipt proving the pages are gone, all while keeping the rest of the library perfectly organized and open for business.

Technical Summary

Here is a detailed technical summary of the paper "Missing-by-Design: Certifiable Modality Deletion for Revocable Multimodal Sentiment Analysis."

1. Problem Statement

Multimodal sentiment analysis (MSA) systems integrate text, audio, and visual data to infer human affect. However, these systems face two critical challenges:

1. Robustness to Missing Modalities: In real-world scenarios, modalities are often missing due to privacy choices, sensor failures, or collection constraints. Models trained on complete data often fail when inputs are incomplete.
2. Verifiable Modality Revocation: With increasing privacy regulations (e.g., GDPR "Right to be Forgotten"), users may request the removal of specific modality data (e.g., audio) from a trained model. Current machine unlearning methods struggle to provide verifiable guarantees that specific modality information has been removed without requiring full model retraining, which is computationally expensive.

The paper addresses the dual need for a model that is robust to missing inputs and capable of performing certifiable deletion of specific modalities while maintaining downstream task utility.

2. Methodology: Missing-by-Design (MBD)

MBD is a unified framework that combines structured representation learning with a certifiable parameter-modification pipeline. It operates in two main stages:

A. Training Phase: Property-Aware Representation & Reconstruction

The goal is to learn embeddings that separate sample-invariant modality characteristics from sample-specific signals, enabling robust fusion and reconstruction.

• Property Embeddings ($P^m$): The model learns a property embedding for each modality that captures sample-invariant priors.
• Decomposition: Input embeddings are decomposed into a sample-specific component ($\Sigma^m_i$) and a sample-invariant component ($\mu^m_i$). Orthogonality and invariance losses ensure these components are distinct.
• Generative Reconstruction: Dedicated generator networks ($G_m$) use available modalities and property embeddings to reconstruct missing channels. This ensures the model can function even when a modality is absent.
• Contrastive Back-Translation: A back-translation network ($B_m$) maps the fused embedding back to the modality space, using Noise-Contrastive Estimation (NCE) to align the reconstructed features with the original distribution.
• Loss Function: The total loss combines supervised task loss, reconstruction loss, property alignment loss, and contrastive regularization.

B. Deletion Phase: Certifiable Weight Surgery

When a user requests the deletion of a modality $m^*$, MBD performs a "surgical" update rather than retraining.

1. Saliency & Importance Proxy:
   • Modality Saliency ($s^{(m^*)}_q$): Computed by aggregating gradients of the reconstruction loss for modality $m^*$ over a calibration batch. High saliency indicates the parameter is critical for that modality.
   • Importance Proxy ($L_q$): A SwiftPrune-inspired metric approximating the expected loss increase if a parameter is removed, avoiding expensive Hessian computations.
2. Candidate Selection: Parameters are selected for modification if they have high modality saliency (to ensure forgetting) but low general importance (to preserve utility).
3. Gaussian Mechanism & Indistinguishability:
   • The selected parameters are modified by either zeroing them out (deterministic) or adding Gaussian noise.
   • The noise scale $\sigma$ is calibrated based on the $\ell_2$-sensitivity of the surgery operator to satisfy a Differential Privacy (DP)-like indistinguishability guarantee.
   • Goal: Ensure the modified model $W'$ is statistically indistinguishable from a hypothetical model $W_{-m^*}$ that was never trained on modality $m^*$.
4. Modality Deletion Certificate (MDC): The process outputs a machine-verifiable artifact containing the deleted modality, modified indices, noise parameters, and a cryptographic hash of the new weights, allowing third parties to verify the deletion.

3. Key Contributions

1. Unified Framework (MBD): The first framework to simultaneously address robustness to missing modalities and certifiable modality-level deletion in sentiment analysis.
2. Property-Aware Decomposition: A novel mechanism to separate invariant modality priors from sample-specific signals, enabling high-fidelity reconstruction of missing channels and stable fusion.
3. Certifiable Surgery Pipeline: A numerically stable approach using saliency-driven selection and Gaussian calibration to produce a Modality Deletion Certificate (MDC), providing mathematical guarantees of deletion without full retraining.
4. Empirical Validation: Demonstrated on standard benchmarks (CMU-MOSI, CMU-MOSEI, IEMOCAP) that MBD achieves state-of-the-art performance under missing data regimes and offers a tunable privacy-utility trade-off.

4. Experimental Results

• Performance with Complete Modalities: MBD outperforms existing baselines (e.g., HyCon, UniMSE, ConFEDE) on all three benchmarks, achieving significant improvements in Accuracy (Acc7, Acc2), F1, and Correlation.
• Robustness to Missing Modalities: Under fixed missing patterns and varying global missing rates, MBD consistently outperforms competitors. For example, on CMU-MOSI with only text available, MBD achieved 88.9% Acc2 compared to 86.2% for the next best method.
• Certified Deletion Efficacy:
  • Privacy: After deleting the audio modality with a conservative privacy budget ($\epsilon_{mod} \le 1$), the Attack Success Rate (ASR) for white-box and black-box adversaries dropped to near-chance levels (~50%).
  • Utility: Despite deletion, the model retained high sentiment accuracy (e.g., Acc2 dropped by only ~1.0–1.5% compared to the full model).
  • Efficiency: The surgical deletion process took 39 seconds on an RTX-3090, compared to 2.9 hours required for full retraining from scratch (a ~270x speedup).
• Ablation Studies: Confirmed that the property embedding pathway and reconstruction module are the primary drivers of robustness, while the unlearning module effectively balances privacy and utility.

5. Significance and Impact

• Privacy Compliance: MBD provides a practical, auditable solution for "Right to be Forgotten" requests in multimodal AI, moving beyond theoretical concepts to a deployable pipeline with machine-verifiable certificates.
• Operational Efficiency: By avoiding full retraining, MBD makes modality revocation feasible for large-scale systems where retraining is prohibitively expensive.
• Robust AI: The framework inherently improves model robustness against missing or corrupted data, making it suitable for real-world deployment where data quality is variable.
• Theoretical Guarantee: The use of DP-inspired indistinguishability bounds provides a rigorous mathematical foundation for claiming that specific modality information has been removed, addressing a major gap in current machine unlearning literature.

In summary, Missing-by-Design represents a significant advancement in privacy-preserving multimodal learning, offering a "surgical" approach to unlearning that is both computationally efficient and mathematically verifiable.