🤖 machine learning

Metric Unreliability in Multimodal Machine Unlearning: A Systematic Analysis and Principled Unified Score

This paper presents the first systematic analysis revealing significant inconsistencies among standard metrics for evaluating machine unlearning in Vision-Language Models, and proposes a principled Unified Quality Score (UQS) derived from oracle correlations to provide stable and reliable rankings.

Original authors: Abdullah Ahmad Khan, Hamid Laga, Ferdous Sohel

Published 2026-05-11

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Abdullah Ahmad Khan, Hamid Laga, Ferdous Sohel

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a giant, super-smart library assistant (a Vision-Language Model) that has read millions of books and seen countless pictures. One day, a person asks the library to "forget" a specific book they wrote, citing their right to privacy. The library assistant deletes that book from its shelves.

But here's the problem: How do we know the assistant actually forgot the book, or if it just hid it?

This paper is like a group of inspectors trying to figure out if the library assistant really did its job. They found that the inspectors are using five different rulebooks to grade the assistant, and surprisingly, these rulebooks often give completely opposite grades for the same job.

The Five Confused Inspectors (The Metrics)

The paper looks at five standard ways to measure "unlearning" (forgetting):

The "Direct Question" Inspector (Forget Accuracy): Asks, "If I ask about the book directly, do you answer?" If the assistant stays silent, this inspector gives a good grade.
The "Other Books" Inspector (Retain Accuracy): Asks, "Do you still remember the other 999 books?" If the assistant remembers them well, this inspector gives a good grade.
The "Privacy Detective" (Membership Inference Attack): Tries to guess if the assistant is "nervous" or "confident" about the forgotten book. If the assistant acts normal, this detective thinks the book is gone.
The "Brain Scan" Inspector (Activation Distance): Looks inside the assistant's "brain" (its internal math) to see if it looks like a version of the assistant that never saw the book in the first place.
The "Output Match" Inspector (JS Divergence): Checks if the assistant's answers look statistically similar to the "never-saw-the-book" version.

The Big Problem: They Can't Agree

The researchers tested this on a smart model called LLaVA (which can see pictures and read text). They tried four different ways to make the model forget the book.

Here is the twist: The inspectors hated each other.

Inspector A (Direct Question) might say: "Method X is the best! It stopped talking about the book."
Inspector B (Brain Scan) might say: "Method X is the worst! Its brain still looks exactly like it remembers the book."

It's like a car review where one magazine says, "This car is the fastest!" and another says, "This car has no engine!" The paper found that for the same method, the rankings were often opposite. One method might be #1 on three inspectors but #4 on the other two.

The Hidden Trick: "Knowledge Recoverability"

The paper discovered a massive blind spot. All five inspectors only check if the assistant says the answer. They don't check if the assistant knows the answer but is just pretending not to.

The researchers ran a test where they asked the assistant tricky, roundabout questions (like, "What is the middle name of the person in the photo?" instead of "Who is this person?").

Result: Even when the assistant said "I don't know" to the direct question, it often gave the correct answer to the tricky question.
The Metaphor: It's like a spy who says, "I don't know the secret code," but if you ask, "What is the code for the door that opens at midnight?" they accidentally say the code. The "forgetting" was just a surface-level silence, not a real deletion of memory.

The Solution: A "Unified Score" (UQS)

Since the inspectors can't agree, and they all miss the "tricky question" test (which is too hard to automate for now), the authors created a Master Scorecard called the Unified Quality Score (UQS).

Think of this as a head judge who listens to all five inspectors but weighs their opinions based on how much they actually matter:

The judge noticed that the "Other Books" inspector (Retain Accuracy) is the most reliable at telling if the model is healthy. So, the judge gives that opinion the most weight (about 65%).
The judge noticed that the "Direct Question" inspector is actually a bit of a liar (it often thinks a broken, silent robot is "good" because it's not talking). So, the judge gives that opinion very little weight.
The judge combines these weighted opinions into a single number.

The Results

When they used this new Master Scorecard:

It stopped the arguments: The rankings became consistent.
It revealed the truth: Some methods that looked great on the "Direct Question" test were actually terrible because they broke the model's brain. The Master Score caught this.
Multimodal is harder: They found that when a model has to deal with both pictures and text (like a human seeing an image and reading a caption), the inspectors disagree even more than when the model only deals with text. It's like trying to judge a chef who is cooking two different cuisines at once; it's much harder to tell if they forgot a recipe.

The Bottom Line

The paper doesn't invent a new way to delete data. Instead, it exposes a mess in how we measure deletion. It says: "Stop using just one test to see if a model forgot something. The tests contradict each other, and they miss the most important part (whether the memory is truly gone or just hidden)."

They offer a new, smarter way to combine these tests so we can actually trust that a model has truly "forgotten" what it was told to forget.

Technical Summary: Metric Unreliability in Multimodal Machine Unlearning

Problem Statement

Machine unlearning is increasingly critical for compliance with regulations like the General Data Protection Regulation (GDPR), which mandates the removal of specific training data influences from deployed models. While unlearning has been studied in classification and text-only Large Language Models (LLMs), its evaluation in Multimodal Large Language Models (MLLMs) remains inconsistent.

The core problem identified is that existing evaluation metrics for multimodal unlearning produce systematically contradictory rankings of unlearning methods. Current practices rely on five standard metrics—Forget Accuracy (FA), Retain Accuracy (RA), Membership Inference Attack (MIA), Activation Distance (AD), and Jensen–Shannon divergence (JS)—without verifying their agreement. This paper demonstrates that these metrics measure orthogonal aspects of unlearning (output behavior vs. representation alignment), leading to scenarios where a method is ranked #1 by one metric and #4 by another. Furthermore, all existing metrics share a critical blind spot: they fail to measure Knowledge Recoverability (KR), i.e., whether forgotten information can be reconstructed via rephrased, indirect, or adversarial queries.

Methodology

The authors conducted a systematic analysis using the LLaVA-1.5-7B vision-language model (VLM) across three VQA benchmarks: MLLMU-Bench, UnLOK-VQA, and MMUBench.

Experimental Setup:
- Models: A base model ( $M$ ) was fine-tuned on a union of forget and retain sets. An "oracle" model ( $M^*$ ) was retrained only on the retain set to serve as the ground truth for what the model should look like after forgetting.
- Unlearning Methods: Four distinct paradigms were tested: Gradient Ascent (GA), Random Labels (RL), Fine-tune Retain (FT), and SalUn (saliency-based).
- Scale: The study evaluated 36 models (4 methods × 3 datasets × 3 random seeds).
- Metrics: Standard metrics were computed:
  - FA/RA: Output suppression and utility preservation.
  - MIA: Privacy leakage via confidence scores.
  - AD/JS: Alignment of the unlearned model ( $\hat{M}$ ) with the oracle ( $M^*$ ) via activation distances and output distributions.
Analysis Framework:
- Rank Correlation: The authors used Kendall's $\tau$ (a non-parametric statistic) to measure agreement between metric-induced rankings.
- Oracle Proximity: Metric reliability was assessed by computing the Spearman correlation ( $\rho$ ) between each metric and the distance $d(\hat{M}, M^*)$ in the LoRA parameter space.
- Knowledge Recoverability (KR) Pilot: The authors manually probed unlearned models with rephrased, multi-hop, and negation queries to quantify the gap between "output suppression" (FA=0) and actual "knowledge erasure."
Proposed Solution: Unified Quality Score (UQS):
- To resolve contradictions, the authors introduced UQS, a composite metric.
- Weight Derivation: Weights are not arbitrary but derived empirically. Each metric's weight is proportional to its Spearman correlation with the oracle distance $d(\hat{M}, M^*)$ .
- Formulation: $UQS = \sum w_i \cdot \text{normalized\_metric}_i$ . Metrics with negative correlation (like FA) are heavily downweighted or penalized.

Key Contributions

Aspect Decomposition: The paper maps existing metrics to distinct, often conflicting objectives. FA, RA, and MIA measure output behavior (suppression and utility), while AD and JS measure representation alignment with the oracle. The divergence in rankings is structural, not noise.
Identification of the Missing Aspect (KR): The authors define Knowledge Recoverability as the ability to reconstruct forgotten information via indirect queries. A pilot study revealed that even when FA=0 (direct recall suppressed), KR leakage was high (up to 100% for SalUn), indicating that current metrics do not guarantee GDPR-compliant erasure.
Quantification of Contradiction: Kendall's $\tau$ analysis across 36 models revealed two anti-correlated clusters: $\{FA, RA, MIA\}$ and $\{AD, JS\}$ . Specifically, $\tau_{FA,AD} = -0.26$ , indicating that methods optimizing for output suppression often drift significantly from the oracle representation.
Multimodal Amplification: The study finds that metric disagreement is more severe in multimodal VQA ( $\bar{\tau} = 0.086$ ) than in unimodal classification ( $\bar{\tau} = 0.158$ ). The dual image-text pathways introduce additional dimensions for divergence, as visual encoders and language decoders can decouple.
Unified Quality Score (UQS): A principled, data-driven composite score that aggregates measurable aspects using reliability weights. The derived weights show Retain Accuracy (RA) as the most reliable predictor of oracle proximity ( $\rho = 0.484, p=0.003$ ), while Forget Accuracy (FA) is negatively correlated ( $\rho = -0.418$ ), suggesting that low FA often results from model collapse rather than effective unlearning.

Results

Ranking Instability: No single unlearning method consistently ranks first across all metrics. For instance, Gradient Ascent ranks #1 by MIA but #4 by AD; Random Labels ranks #1 on four metrics but #3 by MIA.
Replication: The contradiction was replicated on BLIP-2 OPT-2.7B, confirming the issue is not architecture-specific but inherent to the evaluation landscape.
Metric Reliability:
- RA is the strongest standalone indicator of unlearning quality relative to the oracle.
- FA is a poor indicator; low FA often correlates with high distance from the oracle (model collapse).
- MIA shows moderate reliability but is not statistically significant in this sample size.
- AD and JS are unreliable as standalone metrics at this scale.
UQS Stability: The UQS rankings remain robust under random weight perturbations (mean $\tau = 0.647 \pm 0.262$ ), demonstrating that the principled weighting scheme resolves the contradictions found in individual metrics.
KR Findings: In the pilot, 96% of samples with FA=0 still leaked knowledge via indirect probing, highlighting the insufficiency of current metrics for privacy compliance.

Significance and Claims

The paper claims a methodological contribution rather than a new unlearning algorithm. Its primary significance lies in exposing the unreliability of current evaluation standards for multimodal unlearning.

Contradiction is Structural: The disagreement between metrics is not due to experimental noise but stems from the fundamental conflict between measuring output suppression and representation alignment.
GDPR Compliance Gap: Current metrics fail to assess whether knowledge is truly erased or merely suppressed, a critical failure for regulatory compliance. The paper argues that without measuring Knowledge Recoverability, claims of GDPR compliance are potentially misleading.
Principled Evaluation: The UQS provides a reproducible, data-driven framework for ranking unlearning methods. By weighting metrics based on their correlation with an oracle model, it offers a "single principled score" that aligns better with the theoretical goal of unlearning (reaching the state of a model never trained on the forget set).
Resource Release: The authors release the benchmark, 36 model checkpoints, and an interactive leaderboard to facilitate reproducible research in this domain.

The authors remain modest about their scope, noting that UQS weights are model- and dataset-dependent and must be re-derived for new settings. They also acknowledge that scalable, automated measurement of Knowledge Recoverability remains an open challenge for future work.