🤖 machine learning

The Unseen Hand: Manipulating Model Fairness and SHAP with Targeted Identity Re-Association Attacks

This paper introduces Targeted Identity Re-Association (TIRA) attacks, a novel data-agnostic method using probabilistic micro-shuffling and rank-shift perturbations to stealthily manipulate machine learning models, thereby artificially optimizing fairness metrics and completely nullifying protected feature attributions in SHAP explanations.

Original authors: Sannaan Khan, Muhammad U. S. Khan

Published 2026-06-23

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sannaan Khan, Muhammad U. S. Khan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: The "Magic Trick" of AI

Imagine you hire a referee to judge a sports game. To make sure the referee is fair, you have two tools:

The Scoreboard: A list of numbers showing who won and by how much.
The Replay Camera (SHAP): A slow-motion camera that zooms in to show why the referee made a specific call (e.g., "He called a foul because the player was wearing red shoes").

In the world of Artificial Intelligence (AI), these tools are used to check if a model is being fair to different groups of people (like men vs. women, or different ethnicities).

The Paper's Claim: The authors discovered that a "hacker" doesn't need to break the referee's brain or change the game rules. Instead, they can perform a subtle magic trick on the list of names and scores after the game is over. By shuffling the names around just a tiny bit, they can make the scoreboard look perfectly fair and trick the Replay Camera into thinking the referee never cared about the players' identities at all.

The Problem: The "Obvious" Tricks

Previous attempts to trick these fairness checks were like a magician waving a giant, flashing wand.

They would swap huge chunks of data.
The Result: The fairness scores would change, but the "Replay Camera" (SHAP) would still see the mess. It would scream, "Hey! Something weird happened here!" and the trick would be caught.

The Solution: The "Unseen Hand" (TIRA Attacks)

The authors introduce a new family of attacks called TIRA (Targeted Identity Re-Association). Think of this not as a magic wand, but as a micro-surgeon or a gentle breeze.

Instead of making big, obvious swaps, TIRA uses two specific techniques to nudge the data:

Probabilistic Micro-Shuffling (PMiS):
- The Analogy: Imagine a line of people waiting for tickets. The hacker stands in the crowd. Every few seconds, they flip a coin. If it's heads, they quietly swap the person at the front of the line with the person right behind them only if the person behind is from a "disadvantaged" group.
- The Effect: They do this thousands of times, but only a tiny percentage of the time. The line looks mostly the same, but the order has shifted just enough to change the statistics.
Probabilistic Rank-Shift Micro-Perturbation (PRSMP):
- The Analogy: This is like the previous trick, but instead of swapping neighbors, the hacker gently nudges a person a few spots down the line (or up) within a small window. Again, they do this randomly and rarely.
- The Effect: It creates a "drift" in the data that feels natural, like a crowd shifting slightly as people get comfortable, rather than a sudden stampede.

What Did They Achieve?

The paper tested these tricks on real-world data (like predicting diabetes risk or credit approval) and found two major results:

1. The Scoreboard Looks Perfect (Fairness Metrics)
By using these gentle, random shuffles, the hackers could push the "Fairness Score" to look like a perfect 10/10.

Before the attack: The model looked biased (e.g., "It rejects women 20% more often").
After the attack: The model looks perfectly fair (e.g., "It treats everyone exactly the same").
Key Point: The actual predictions the AI made didn't change; only who those predictions were assigned to changed.

2. The Replay Camera is Fooled (SHAP)
This is the most dangerous part. When the authors ran the "Replay Camera" (SHAP) on the hacked data:

Before: The camera clearly showed, "The model is looking at Gender to make decisions."
After: The camera showed zero influence from Gender. It looked like the model didn't care about gender at all.
The Takeaway: The attack successfully hid the "footprints" of the manipulation. The auditor sees a clean, fair model and thinks, "Everything is fine," when in reality, the model's fairness has been artificially manufactured.

Why Does This Matter?

The paper argues that we have been too trusting of these "post-game" checks.

We assume that if the Scoreboard says "Fair" and the Replay Camera says "No Bias," then the AI is safe.
This paper proves that a clever attacker can manipulate the names on the list to make both tools lie.

The "Knobs" of Control

The authors emphasize that this isn't a blunt instrument. They can turn "knobs" (parameters) to control the attack:

How often to swap? (Probability)
How far to move someone? (Rank shift)
How many times to do it? (Iterations)

This allows an attacker to make the model look slightly fair or perfectly fair, all while keeping the changes so small that no one notices the "Unseen Hand" at work.

Summary

The paper warns us that AI fairness audits are fragile. Just because an AI model passes a fairness test and looks explainable, it doesn't mean it's actually fair. A subtle, probabilistic shuffling of data identities can trick our best tools into seeing a perfect model where a biased one actually exists.

Technical Summary: The Unseen Hand: Manipulating Model Fairness and SHAP with Targeted Identity Re-Association Attacks

1. Problem Statement

As machine learning models become increasingly opaque and influential in critical societal domains, the reliance on algorithmic fairness audits and post-hoc explainability tools (such as SHAP) has grown to ensure accountability. However, this paper identifies a critical vulnerability: these auditing mechanisms themselves are not robust against adversarial manipulation.

Prior research has demonstrated that "fooling" strategies can manipulate fairness metrics, but these existing methods often rely on overt perturbations or require access to the model's internal data distribution. Consequently, they leave detectable artifacts—such as significant drops in fairness scores or anomalous SHAP attribution values—that trigger suspicion during audits. There is a lack of subtle, data-agnostic, post-hoc manipulation techniques capable of altering a model's perceived fairness and undermining explanation tools without leaving forensic footprints.

2. Methodology

The authors propose a novel family of attacks termed Targeted Identity Re-Association (TIRA). These attacks operate under a black-box threat model where the adversary has access only to the model's output scores and the protected attributes of the input instances, but no access to the model architecture, training data, or decision logic.

The core framework involves:

Score Sorting: Obtaining prediction scores from the black-box model and sorting them in descending order.
Identity Re-Association: Iteratively and probabilistically modifying the association between the sorted scores and the identities (specifically the protected attributes) without altering the scores themselves.
Output Re-Assembly: Reconstructing the dataset with the shuffled identities to recalculate fairness metrics and SHAP values.

The paper formalizes two specific algorithms within the TIRA family:

Probabilistic Micro-Shuffling (PMiS): This algorithm applies localized, adjacent swaps. In each iteration, it examines adjacent pairs in the sorted list. If a disadvantaged individual (protected attribute = 0) is adjacent to an advantaged individual (protected attribute = 1), it probabilistically swaps their identities with a probability $p_{swap}$ . This creates a cumulative, gradual drift in the distribution of protected attributes relative to scores.
Probabilistic Rank-Shift Micro-Perturbation (PRSMP): This algorithm introduces a broader locality by allowing rank shifts within a small window. It targets disadvantaged individuals and probabilistically shifts their identity to another disadvantaged individual within a predefined rank window ( $k_{max}$ ) with probability $p_{shift}$ . Like PMiS, this relies on multiple iterations to accumulate significant rearrangements while maintaining subtlety.

3. Key Contributions

The paper makes four primary contributions:

Formalization of TIRA Attacks: The introduction of Targeted Identity Re-Association as a novel, post-hoc, data-agnostic, and probabilistic method for subtly manipulating model outputs.
Empirical Validation of Fairness Manipulation: Demonstration that TIRA attacks can significantly alter AIF360 fairness metrics (e.g., Equal Opportunity, Demographic Parity, Disparate Impact) across diverse model-dataset configurations, guiding them toward desired ideal thresholds.
Impact on SHAP Attribution: Illustration of how TIRA attacks confound SHAP-based explanations. The attacks successfully reduce the SHAP attribution for protected features to effectively zero, creating a false impression that the model does not rely on these features.
Comparative Benchmarking: A demonstration that TIRA attacks offer finer granularity and greater control compared to existing overt attacks (such as DomSwap and MixSwap), achieving comparable or superior shifts in fairness metrics with significantly higher stealth.

4. Results and Analysis

The authors evaluated TIRA attacks on the Bangladeshi Diabetes Dataset and the German Credit Dataset using both Logistic Regression and Neural Network models.

Fairness Metrics: The results showed that TIRA attacks consistently pushed fairness metrics toward ideal values (e.g., reducing Demographic Parity differences and Disparate Impact ratios closer to 1). In many cases, PMiS and PRSMP achieved more precise tuning of these metrics compared to baseline shuffling attacks. For instance, on the Diabetes dataset, PMiS reduced the Equal Opportunity difference from a baseline of 0.09 to -0.07, while other attacks showed less consistent or less extreme shifts.
SHAP Attribution: Post-attack analysis revealed that the SHAP values for the protected features were effectively zero. This indicates that the attacks successfully obscured the model's reliance on protected attributes from the explanation tool, a significant improvement over prior work where detectable attribution values often remained.
Stealth and Control: The study highlighted the "fine-grained control" offered by tunable parameters ( $p_{swap}$ , $p_{shift}$ , $k_{max}$ , and iterations $T$ ). By adjusting these, an adversary can scale the manipulation from almost imperceptible changes to pronounced alterations, allowing for a trade-off between the magnitude of the fairness shift and the stealth of the attack.

5. Significance and Claims

The paper argues that the fairness assessment itself is brittle. The ability to alter a model's perceived fairness and explanation profiles through subtle, non-intrusive changes to output rankings challenges the assumption that trustworthy AI can be achieved solely through post-hoc explanation methods.

The authors claim that TIRA attacks represent a "dual-pronged threat" not previously achieved:

They can fool AIF360 audits by pushing metrics to near-ideal values.
They can fool XAI audits by reducing SHAP attribution to effectively zero.

The significance of this work lies in exposing the insufficiency of current auditing frameworks. The enhanced stealth of TIRA attacks suggests that the absence of a strong attribution signal does not necessarily equate to fairness or a lack of adversarial perturbation. Consequently, the paper concludes that there is an urgent need to develop integrity-based evaluation methodologies that can detect inconsistencies between a model's outputs and its explanations, rather than relying solely on the metrics and attribution values provided by current tools.