🤖 AI

CoAX: Cognitive-Oriented Attribution eXplanation User Model of Human Understanding of AI Explanations

This paper introduces the CoAX user model, which employs cognitive modeling to analyze and simulate human reasoning strategies when interpreting AI explanations for tabular data, thereby offering a scalable method to diagnose why current XAI methods often fail to improve user understanding and to guide the development of more effective explanations.

Original authors: Louth Bin Rawshan, Zhuoyu Wang, Brian Y. Lim

Published 2026-05-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Louth Bin Rawshan, Zhuoyu Wang, Brian Y. Lim

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, but mysterious, robot friend who makes decisions for you—like telling you if a wine is "good" or "bad," or if a loan application should be approved. You want to trust this robot, so it gives you a little note explaining why it made that choice. This is called Explainable AI (XAI).

But here's the problem: even with the note, people often still get confused or make the wrong guesses about what the robot will do next. It's like the robot is speaking a language you don't quite understand, even though the words are right there.

This paper, titled CoAX, is like a "translator" and a "debugger" for how humans think when they try to understand these robot notes. The researchers built a virtual human (a computer program that acts like a person) to figure out exactly how our brains process these explanations, and why we sometimes get it wrong.

Here is the story of their discovery, broken down simply:

1. The Problem: The "Black Box" and the Confused User

The researchers noticed that even though scientists have invented many ways to explain AI decisions, real people still struggle to use them. Some explanations make people worse at predicting what the AI will do. The authors asked: "Why? What is happening inside the human brain when they look at these charts and numbers?"

2. The Solution: Building a "Virtual Human" (CoAX)

Instead of just guessing, the researchers built CoAX (Cognitive-Oriented Attribution eXplanation). Think of CoAX as a video game character that is programmed to think exactly like a real human.

How it works: The researchers first watched real people (in a "formative study") and asked them to "think out loud" while looking at AI explanations. They noticed people used specific mental tricks, or "strategies," to make sense of the data.
The Strategies:
- The "Ignore the Rest" Strategy: Some people just pick one or two features they think matter (like "Alcohol content") and ignore everything else.
- The "Highlighter" Strategy: When given a chart showing which features are "important," people focus only on the biggest bars and try to remember similar past cases.
- The "Math Sum" Strategy: When given a detailed breakdown (positive and negative scores), people try to add up the numbers in their heads to see which side wins.
The Model: CoAX takes these strategies and turns them into math. It simulates a human who has a limited memory (they forget old examples) and limited attention (they can only look at a few things at once).

3. The Experiment: Testing the Virtual Human

The researchers then ran a massive experiment with over 300 real people. They asked these people to guess what an AI would predict for different scenarios (like wine quality or income levels) under three conditions:

No Help: Just the data.
Simple Help: A list of "Important" features.
Detailed Help: A list of features with positive and negative scores (Attribution).

They compared the real humans' answers to the Virtual Human's answers.

The Big Discovery:
The Virtual Human (CoAX) was much better at predicting what real humans would do than standard computer programs (like typical Machine Learning models) were.

Standard computer models just look for patterns in the data.
CoAX looks for reasoning patterns. It realized that humans aren't just calculating; they are using mental shortcuts, getting confused by too much info, or misinterpreting the "importance" of a feature.

4. What They Found About Human Thinking

Using their Virtual Human, they discovered some interesting things about how we fail to understand AI:

The "Math Sum" is Best: When people were given detailed "Attribution" explanations (showing exactly how much each factor pushed the decision up or down), they tended to use a "summing" strategy. This worked the best. It's like adding up the pros and cons on a piece of paper.
The "Simple List" is Tricky: When people were given a simple list of "Important" features (without showing if they were good or bad), they got confused. Some tried to guess the direction of the influence, and others just ignored the list entirely and went back to their old habits.
Confusion Leads to Ignoring: If an explanation was too confusing, people would just throw it away and guess based on what they remembered from the past, effectively ignoring the AI's help.

5. Why This Matters (The "Virtual Lab")

The coolest part of this paper is that once they built this Virtual Human, they could use it as a simulation lab.

Instead of hiring 300 people, paying them, and running weeks of experiments to test a new idea, researchers can now just "ask" the Virtual Human.

Example: "What happens if we show 9 features instead of 5?" The Virtual Human can simulate this instantly and tell you, "Oh, people will get overwhelmed and their accuracy will drop."
This allows designers to debug AI explanations before they ever show them to a real person. They can fix the confusing parts in the computer model first.

The Bottom Line

The paper argues that to make AI explanations better, we need to stop treating humans like simple data processors. We need to understand that humans have limited memory, selective attention, and specific mental habits.

By building a Virtual Human that mimics these habits, we can figure out why current AI explanations fail and design better ones that actually help people understand the robot's mind, rather than just confusing it further. It's like tuning a radio to the right frequency so the human brain can finally hear the signal clearly.

1. Problem Statement

Despite significant advancements in Explainable AI (XAI), a critical gap remains: many XAI techniques fail to improve actual user understanding or decision-making. Previous studies have shown that explanations can sometimes confuse users, lead to over-reliance, or fail to help users detect model errors.

The Core Issue: Existing research often relies on surface-level behavioral metrics (e.g., accuracy) or conjecture to explain why users struggle. There is a lack of rigorous, mechanistic models that explain the cognitive processes and reasoning strategies users employ when interpreting XAI.
The Consequence: Without understanding the underlying cognitive mechanisms (e.g., how users process feature importance vs. feature attribution), it is difficult to design effective explanations or debug why specific XAI methods fail.

2. Methodology

The authors propose CoAX, a cognitive model that integrates Instance-Based Learning (IBL) theory with cognitive constraints (memory decay, attention limits) to simulate human reasoning on XAI. The research follows a five-step user-centered design process:

A. Formative User Study (Strategy Elicitation)

Goal: Identify how humans actually reason when using XAI.
Participants: 20 undergraduate students.
Task: "Forward Simulation" (predicting the AI's output given an input instance and an explanation).
Conditions: No XAI, Feature Importance (non-negative scores), and Feature Attribution (signed scores, e.g., SHAP/LIME).
Method: Think-aloud protocols analyzed via grounded theory.
Outcome: Identification of four distinct reasoning strategies:
1. Sensitive-features categorization: Focusing on a fixed subset of attributes believed to be important, ignoring others.
2. Salient-features categorization: Dynamically selecting the top- $k$ features with the highest importance scores and recalling similar past instances.
3. Attribution sum: Summing signed attribution scores (or estimating them) to weigh evidence for/against a label.
4. Importance categorization: Misinterpreting importance scores as direct attribute values for pattern matching (a flawed strategy).

B. CoAX Cognitive Model Implementation

The model simulates human decision-making using IBL:

Memory: Stores past exemplars (instances) with activation levels based on recency and frequency.
Retrieval: Retrieves similar past instances based on Euclidean distance, weighted by activation.
Strategy Extensions: The base IBL model is extended to implement the four identified strategies:
- Sensitive/Salient: Modifies the distance metric to focus only on specific subsets of features.
- Attribution Sum: Calculates a log-odds score by summing weighted feature contributions.
- Importance Categorization: Treats importance scores as feature values rather than weights.
Parameters: Includes distance sensitivity ( $\alpha$ ), retrieval threshold ( $\rho$ ), number of features attended ( $k$ ), and feature-class sensitivity ( $\zeta$ ).

C. Summative User Study (Validation)

Participants: 335 participants (Prolific).
Design: Between-subjects for XAI type (None, Importance, Attribution) and within-subjects for testing condition (With/Without XAI).
Datasets: Wine Quality, Forest Cover Type, Adult Income (binary classification, 5 attributes).
Goal: Validate CoAX by fitting the model to human decision data and comparing its predictive power against standard Machine Learning (ML) proxies (Decision Trees, KNN, MLP).

D. Virtual Hypothesis Testing

Goal: Use the validated CoAX model to run "virtual experiments" that would be too costly or slow to conduct with real humans.
Variables: Number of training instances, number of attributes, and different XAI algorithms (SHAP, LIME, Integrated Gradients, Input Gradients).

3. Key Contributions

Taxonomy of Reasoning Strategies: Empirically elicited and categorized four distinct cognitive strategies users employ when interpreting attribution-based XAI, including specific failure modes (e.g., misinterpreting importance scores).
CoAX Cognitive Model: Developed a formal cognitive model that simulates these strategies using Instance-Based Learning, capturing the procedural logic of human reasoning rather than just statistical correlations.
Superiority over ML Proxies: Demonstrated that CoAX fits human decision data significantly better than standard ML proxies (DT, KNN, MLP), proving that human reasoning in XAI contexts is driven by specific cognitive heuristics, not just data patterns.
Virtual Experimentation Framework: Established a pipeline to use CoAX for hypothesis generation and debugging XAI usability, reducing the reliance on expensive human-subject studies for early-stage prototyping.

4. Key Results

Model Fit: CoAX achieved a much lower Negative Log-Likelihood (NLL) and Bayesian Information Criterion (BIC) compared to ML baselines, indicating it captures the underlying human decision process more accurately.
Strategy Effectiveness:
- Attribution Sum (using signed scores) was the most effective strategy, leading to the highest prediction accuracy.
- Importance XAI (non-negative scores) led to diverse and often sub-optimal strategies, including the flawed "Importance categorization."
- Learning Transfer: Participants trained with Attribution XAI retained the "Attribution sum" strategy even when XAI was removed, whereas those trained with Importance XAI often reverted to "Sensitive-features" strategies.
Impact of XAI Type:
- Attribution XAI consistently outperformed Importance XAI and No XAI.
- Feature Count: Increasing the number of attributes degraded performance for "No XAI" and "Importance XAI" (information overload) but was mitigated by "Attribution XAI."
Virtual Experiments: The model successfully simulated how increasing training instances improves understanding (plateauing effect) and how different XAI algorithms (e.g., LIME vs. SHAP) impact user performance, validating the model's utility for scalable testing.

5. Significance and Implications

Debugging XAI: CoAX provides a diagnostic tool to identify why an explanation fails (e.g., does it trigger a flawed "Importance categorization" strategy?).
Design Recommendations:
- UI Design: Designers should encourage "Attribution sum" strategies (e.g., by showing signed values) and discourage "Importance categorization" (e.g., by avoiding the presentation of importance scores as raw values).
- Training: Tutorials should explicitly teach users how to aggregate signed attributions rather than just looking at the "most important" features.
Cost Reduction: By validating hypotheses with virtual humans (CoAX) before running large-scale user studies, researchers can iterate XAI designs more rapidly and cost-effectively.
Future Directions: The framework can be extended to other modalities (images, text), different decision tasks (trust, reliance), and diverse user populations to move beyond "one-size-fits-all" XAI.

In summary, CoAX bridges the gap between cognitive science and XAI, offering a mechanistic understanding of human-AI interaction that moves beyond "does it work?" to "how does the user think?"