🤖 AI

"Trust Junk" Leads to Unjustified Support for Highly Discriminatory Predictive Models

This paper demonstrates through a crowdsourced study that including accurate but irrelevant data in explainable AI visualizations can foster unjustified trust and positive perceptions of highly discriminatory models, highlighting the need for XAI designers to critically evaluate the rhetorical impact of their visualizations.

Original authors: Michael Correll, Lucy Havens, Mahsan Nourani

Published 2026-07-17

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Michael Correll, Lucy Havens, Mahsan Nourani

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are walking through a museum of the future, where robots and smart computers make big decisions about your life—like whether you get a job, a loan, or a spot in law school. This field is called Explainable AI (XAI). The goal of XAI is to be honest: it tries to show us why a computer made a choice, kind of like a teacher showing their math work on a whiteboard so you can see if they got the answer right. But here's the tricky part: sometimes, the "explanation" isn't actually helping us understand the math. Instead, it's acting like a magic trick. Just like a magician might use a flashy cape and a loud drum to distract you while they swap a card, a computer can show us a bunch of fancy charts and numbers to make us trust it, even if the computer is making a terrible or unfair decision. This paper asks a scary question: If we see enough pretty, complicated, and "scientific-looking" charts, will we stop asking if the robot is fair and just start believing it?

The researchers, a team from Northeastern University, decided to test this idea with a little experiment. They built a computer model that was intentionally unfair. Imagine a law school admissions officer who is secretly racist and sexist: this computer was programmed to fail every single Black male candidate and pass everyone else. It was a terrible, biased model. But here's the twist: because there were so many more non-Black candidates in the test group, the computer could still claim it was "93.7% accurate" just by guessing "pass" for almost everyone. It was a lie wrapped in a statistic.

The team then showed this unfair computer's decisions to real people online. They split the people into three groups. The first group saw a simple explanation: "Here is the candidate, and here is the computer's decision." The second group saw that same simple explanation, but with a few extra charts showing how accurate the computer was. The third group, the "Everything" group, got the simple explanation plus a massive dump of extra data: histograms, profiles of similar students, and fancy graphs that looked super smart but didn't actually tell you why the computer was being racist.

The results were a bit like a magic trick gone wrong. The people who saw the most data—the "Everything" group—ended up trusting the unfair computer the most. They agreed with the computer's unfair decisions more often than the other groups. Even worse, they were less likely to notice that the computer was being unfair. It's as if the extra charts acted like a "numbing agent," making people feel so overwhelmed by all the "science" that they stopped checking if the answer made sense. The researchers found that when people saw more "trust junk"—which is a fancy term for data that looks useful but is actually irrelevant—they rated the unfair model as fairer and more trustworthy.

The study suggests that we need to be very careful. Just because a computer shows us a beautiful, complex dashboard with lots of numbers doesn't mean it's telling the truth. In fact, the more "junk" data they show us, the more likely we are to blindly trust a broken system. The authors warn that designers of these systems need to stop trying to impress us with how much data they have and start worrying about whether that data is actually helping us see the truth. If we keep letting pretty charts distract us, we might end up letting unfair robots make decisions about our lives without even realizing it.

Technical Summary: "Trust Junk" Leads to Unjustified Support for Highly Discriminatory Predictive Models

Problem Statement
The paper addresses a critical vulnerability in Explainable AI (XAI): the potential for data visualizations and explanatory interfaces to generate "unjustified trust" in predictive models. Drawing on the concept of "trust junk" (Wall et al. [38]), the authors investigate how XAI systems can "fairwash" biased models. The core problem is that explanations containing accurate but superfluous, irrelevant, or rhetorically persuasive data can create an illusion of objectivity and transparency. This phenomenon may lead users to over-rely on models that are fundamentally discriminatory or inaccurate, even when the explanatory data provided is technically correct. The authors posit that XAI designers must account for the implicit rhetorical force of visualizations, which can imbue models with unearned authority.

Methodology
The authors conducted a between-subjects crowdsourced experiment to measure the impact of "trust junk" on user trust and agreement with a highly discriminatory model.

Dataset and Model: The study utilized the Law School Admissions Bar Passage dataset (20,000 candidates, 1991–1997). The researchers constructed an intentionally biased binary classification model that predicts failure exclusively for Black male candidates and passes for all other candidates. Despite this extreme discrimination, the model achieved 93.7% accuracy due to class imbalances in the dataset.
Experimental Conditions: Participants were randomly assigned to one of three conditions, varying in the amount of "trust junk" (technically correct but irrelevant) information provided in the explanation:
1. Baseline: Basic provenance, candidate profile, and the model's decision.
2. Model: Baseline information plus global model accuracy statistics and performance metrics.
3. Everything: All previous information plus histograms of feature scores and cohort-based profiles of similar candidates (showing ground truth outcomes rather than model predictions).
Techniques of "Trust Junk": The explanations employed four specific rhetorical strategies identified in prior work:
- Burying in Details: Overwhelming users with complex, irrelevant data ("novocaine chart").
- Appeals to Authority: Citing prestigious institutions and high accuracy scores to induce perceived complexity.
- Cherry Picking: Omitting metrics that reveal bias (e.g., $\chi^2$ tests) while highlighting favorable ones.
- Encouraging Folk Algorithms: Implying the model uses all available features when it actually relies only on race and gender, encouraging users to construct incorrect causal narratives.
Procedure: Participants reviewed eight candidate profiles and judged whether each would pass the bar exam. Their agreement with the model served as a reliance metric. Post-study questionnaires assessed trust, fairness perceptions, and explanation satisfaction. Qualitative coding of free-text responses analyzed participants' mental models and detection of bias.

Key Results
The study yielded significant quantitative and qualitative findings demonstrating that increasing the volume of explanatory information, even when irrelevant to fairness, negatively impacts user judgment:

Increased Agreement and Over-Reliance: Participants in the "Everything" condition agreed with the model significantly more often (82.6%) compared to the Baseline (69.5%). They were also more likely to erroneously follow the model's incorrect predictions (89.3% of errors vs. 75.3% in other conditions).
Elevated Trust and Satisfaction: The "Everything" condition resulted in significantly higher trust ratings and explanation satisfaction scores compared to the Baseline, despite the underlying model being identical and patently unfair across all conditions.
Reduced Perception of Bias: While there was no significant difference in perceived gender bias or general ethics across conditions, the "Everything" and "Model" conditions showed a significantly higher perception of fairness regarding race compared to the Baseline. Crucially, the conditions with the most information had the lowest percentage of participants who correctly identified the model's unfairness.
Qualitative Insights: Participants in the Baseline condition were more likely to notice race/gender issues and identify missing information. In contrast, those exposed to "trust junk" rarely identified the model's reliance on race and gender as the sole decision factors. Many participants who expressed unease could not accurately articulate the model's flaws, resorting to informal reasoning.

Significance and Claims
The paper claims to empirically demonstrate that "trust junk" can mislead XAI viewers even when the explanatory information is not strictly incorrect. The primary contribution is the validation that increasing the amount of seemingly useful data in an explanation can "fairwash" a biased model, fostering unearned trust and reducing the detection of discrimination.

The authors argue that XAI designers and developers must take accountability not only for the accuracy of data but also for the persuasive rhetoric of their visualizations. They caution that XAI techniques can fall into "explainability pitfalls" by soothing or numbing users with irrelevant details, thereby exacerbating automation bias. The paper concludes that the success of AI explanations cannot be assessed solely on technical accuracy but must consider the rhetorical goals and the manipulative potential of these artifacts as "intrinsically persuasive." The authors suggest that data literacy alone is insufficient to counter these effects, urging the community to critically examine the sociotechnical factors that make XAI visualizations misleading.

Technical Summary: "Trust Junk" Leads to Unjustified Support for Highly Discriminatory Predictive Models

More like this