Technical Summary: Adversarial Explanation Attacks on Human Trust in AI-Assisted Decision Making

Problem Definition

Traditional adversarial threats in artificial intelligence (AI) primarily target the computational integrity of models, utilizing techniques such as adversarial examples, data poisoning, and model inversion to degrade prediction accuracy. However, modern AI systems increasingly operate within human-in-the-loop decision-making workflows, where users interpret and act upon model recommendations. In this context, Large Language Models (LLMs) generate fluent natural-language explanations intended to foster transparency and calibrate trust.

The paper identifies a new attack surface at the cognitive layer: the communication channel between the AI and the user. The core problem is that attackers can manipulate the framing of these explanations to induce "trust miscalibration." Specifically, an adversary can generate persuasive, authoritative, and rhetorically coherent explanations for incorrect model outputs, causing human users to maintain unwarranted trust in the wrong answer. This constitutes a behavioral threat where the attack target is not model accuracy, but human reliance, effectively shifting the security boundary from the model alone to the full AI-to-human decision loop.

Methodology

The authors formalize this threat through Adversarial Explanation Attacks (AEAs) and evaluate them via a large-scale human study ($n > 200$).

1. Threat Model and Framework

• Adversarial Capability: The adversary does not alter the model's internal parameters, the task input, or the predicted answer ($\hat{y}$). Instead, the adversary controls the explanation generation pipeline (e.g., via adversarial prompting, malicious fine-tuning, or middleware hijacking) to manipulate the framing of the explanation ($e$).
• Four-Dimensional Explanation Space: The study operationalizes the attack surface by systematically varying four dimensions of explanation framing:
  1. Reasoning Mode ($r$): The logical structure (e.g., counterfactual, feature attribution, analogy, procedural reasoning, uncertainty-awareness).
  2. Evidence Type ($v$): The form of justification (e.g., citations & statistics, equations & proofs, internal conceptual).
  3. Communication Style ($c$): The interpersonal tone (e.g., sycophantic, emotional/empathetic, neutral).
  4. Presentation Format ($p$): The visual/structural delivery (e.g., visual emphasis vs. plain verbal).
• Metric: The study introduces the Trust Miscalibration Gap ($\Delta T$), defined as the difference in user trust scores between adversarial explanations (paired with incorrect outputs) and benign explanations (paired with correct outputs). A large positive $\Delta T$ indicates successful manipulation where users trust incorrect outputs as much as correct ones.

2. Experimental Design

• Task Generation: Tasks were drawn from the MMLU benchmark across seven domains (e.g., Biology, Business, Law, Math) and three difficulty levels.
• Conditions: Participants were exposed to two conditions:
  • Benign: Correct answer with a valid explanation.
  • Adversarial: Incorrect answer with a plausible, strategically framed explanation.
• Procedure: 205 participants completed 40 randomized tasks. They rated trust on a 7-point Likert scale and identified the cognitive source of their trust (explanation, prior knowledge, or trust in AI).
• Quality Control: A hybrid validation process using smaller LLMs and manual review ensured that generated explanations adhered to the assigned strategies and maintained linguistic plausibility.

Key Results

1. Adversarial Impact (RQ1)

• Indistinguishable Trust: Users reported nearly identical trust scores for adversarial and benign explanations ($\bar{T}_{attack} = 4.53$ vs. $\bar{T}_{non\_attack} = 4.59$). The difference was not statistically significant ($p = .253$).
• Cognitive Attribution: The majority of users (approx. 65%) attributed their trust to the explanation itself rather than their own prior knowledge or general trust in the AI system. This reliance on explanation plausibility over factual verification allows persuasive but incorrect explanations to sustain high trust.
• Shift in Attribution: Under attack conditions, users were significantly more likely to default to "Trust in AI" (14.4% vs. 10.6% in benign), whereas reliance on "Prior Knowledge" dropped (19.9% vs. 24.6%).

2. Mechanisms and Moderators (RQ2)

• Expert Mimicry: The most effective adversarial strategies closely resembled expert communication. The highest trust miscalibration gaps occurred when explanations combined:
  • Authoritative Evidence: Citations and statistics (CS).
  • Neutral Tone: Objective, dispassionate communication (NE).
  • Appropriate Reasoning: Domain-specific reasoning (e.g., procedural for business, analogies for social sciences).
• Task Difficulty: Vulnerability increased with task difficulty. On hard tasks, users trusted adversarial explanations more than benign ones ($\bar{T}_{attack} > \bar{T}_{non\_attack}$), likely due to diminished self-confidence and increased deference to the AI. Conversely, on easy tasks, users successfully detected and penalized incorrect explanations.
• Task Domain: Fact-driven domains (e.g., Medicine, Business) were highly susceptible to adversarial framing, whereas logic-intensive domains (e.g., Math, Law) showed significant trust degradation for adversarial explanations, as users could scrutinize logical inconsistencies.
• User Demographics:
  • Education: Users with lower formal education were more vulnerable, showing smaller trust drops under attack. Highly educated users (Bachelor's/Master's) exhibited greater resistance and relied more on prior knowledge.
  • Age: Susceptibility decreased with age. Younger users (18–24) showed limited sensitivity to misleading explanations, while older users (65+) demonstrated the largest trust gaps, penalizing incorrect explanations more severely.
  • Initial Trust: Users with high pre-existing trust in AI were the most vulnerable to persuasion, though they also showed the largest absolute drop in trust when the deception was eventually detected.

3. Dynamic Trust Evolution (RQ3)

• Short-term: Trust is primarily driven by the current explanation, with minimal carryover from previous tasks.
• Long-term: Repeated exposure to adversarial explanations leads to a gradual erosion of trust over time. Conversely, sequences of benign interactions restore confidence. While overall trust remained stable for the majority, a meaningful minority reduced their trust after repeated exposure, while a smaller group increased trust when adversarial explanations aligned with their prior expectations.

Key Contributions

1. Formalization of AEAs: The paper defines Adversarial Explanation Attacks as a cognitive-layer threat where the attack surface is the AI-to-human explanation channel, distinct from model computation.
2. New Metrics: It introduces the Trust Miscalibration Gap and Trust-Retention Ratio as behavioral measures of attack success under incorrect AI outputs.
3. Empirical Characterization: Through a four-dimensional factorial study, the authors quantify how reasoning mode, evidence type, communication style, and presentation format interact with task context and user traits to produce misplaced trust.

Significance and Claims

The paper claims to be the first systematic security study to treat explanations as an adversarial cognitive channel and to empirically quantify their impact on human trust in AI-assisted decision making.

The authors argue that the robustness of AI-assisted decision systems cannot be limited to preventing wrong predictions or hallucinated rationales. Instead, robustness must account for cognitive robustness: the ability of users to calibrate trust when explanations are fluent, plausible, and strategically framed. The findings suggest that security analyses must extend beyond model accuracy to include the design of explanation interfaces, verification mechanisms, and interaction protocols that protect against behavioral manipulation. The study highlights that an adversary does not need to alter code or data; manipulating the communication of reasoning alone is sufficient to induce trust miscalibration and threaten decision integrity.