🤖 AI

Addressing the Selection Problem in Explainable AI

This paper identifies the "selection problem" in Explainable AI, where users struggle to choose effective explanation techniques due to siloed research, and proposes a multi-agent LLM orchestration tool to automatically translate user queries into the appropriate XAI method.

Original authors: Claire Vlases, Katelyn Morrison

Published 2026-08-25

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Claire Vlases, Katelyn Morrison

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the rapidly evolving world of artificial intelligence, a quiet frustration has taken hold among the people who build and use these systems. As machines learn to make decisions about everything from medical diagnoses to loan approvals, a field known as explainable AI has emerged to help humans understand how those decisions are made. The goal is simple: to provide a clear reason for a machine's output, turning a black box into something transparent. However, the current landscape is cluttered with dozens of different methods for generating these reasons. Some methods highlight which parts of an image the computer looked at, while others show how a small change in the input would alter the result, or they simply retrieve similar past examples to show how the system usually behaves. Each method answers a different kind of question about the machine's logic. The problem is that while researchers have built a vast toolkit of these explanation methods, the people who need to use them are often left standing in front of a locked door, unsure which tool to pick or how to ask for the right kind of help.

A team of researchers at Carnegie Mellon University has identified this confusion not just as a user error, but as a fundamental flaw in how these systems are designed. They call it the "selection problem." In their view, the current way we interact with explainable AI forces users to translate their own natural confusion into a specific technical command before they can see any explanation. Imagine a person who is unsure why a machine made a certain choice; they might think, "I want to know what would happen if I changed this one detail," or "I need to see an example of a similar case." In the current systems, the user must know that the first thought requires a specific type of explanation called a counterfactual, and the second requires an exemplar method. They must then find the correct button or menu option to select that specific technique. The researchers argue that expecting a regular person to know the technical names and capabilities of these different explanation methods is an unfair and unrealistic demand. Because users cannot reliably match their own questions to the correct technical tool, they often end up with explanations that do not actually answer what they wanted to know, leading to confusion or misplaced trust in the machine.

To prove this point, the authors break down the interaction into a logical sequence. They start by noting that an artificial intelligence model is essentially a function that takes an input and produces an output. An explanation is simply a process that takes that same input and model to produce a reason. Crucially, different explanation processes are designed to answer different structural questions. A user's uncertainty at any given moment is a specific state of mind, a gap in their understanding that they want to fill. The researchers show that for any specific gap in understanding, only a small subset of the available explanation methods would actually be helpful. Yet, in the systems we use today, the user is the one who must bridge the gap between their natural language question and the technical method. The system does not ask, "What are you trying to understand?" It waits for the user to select a tool. Since most people do not have the technical training to know which tool fits which question, they often pick the wrong one. The result is a system that provides an answer, but not the right answer for the user's specific need.

The paper proposes a structural solution to fix this broken link: a system that handles the translation for the user. Instead of asking the user to choose a tool, the new design asks the user to simply speak or type their question in plain English. The system then uses a sophisticated software tool, built on large language models, to listen to the user's uncertainty and automatically route it to the correct explanation method. This approach shifts the burden of translation from the human to the machine. The system accepts the user's natural language, figures out what kind of question is being asked, and then selects the appropriate explanation technique to provide the answer. This allows for a more fluid interaction where the user does not need to know the technical vocabulary of the field. The researchers illustrate this with a prototype designed for medical imaging. In this example, a doctor could type a question like, "I'm not sure why the system flagged this scan," and the system would automatically determine that the doctor needs a specific type of visual explanation, then generate that specific view without the doctor ever needing to know the name of the technique or where to find it.

The researchers are careful to frame this as a proposal and a formalization of a problem, rather than a solved crisis. They present their idea as a logical necessity based on the current limitations of user interfaces, suggesting that the failure of explainable AI to gain widespread trust is partly due to this interface mismatch. They do not claim to have built a perfect, finished product, but rather a blueprint for how future systems should be constructed. By moving the task of matching questions to tools from the user to the system, they argue that we can create a more effective and trustworthy relationship between humans and artificial intelligence. The work suggests that the path forward lies not in building more explanation tools, but in building better ways for people to ask for them.

Technical Summary: Addressing the Selection Problem in Explainable AI

Problem Definition: The Selection Problem
The paper identifies a critical barrier in the adoption of Explainable AI (XAI): the "selection problem." Despite a proliferation of XAI techniques (e.g., saliency maps, counterfactuals, exemplars), user studies indicate that available explanations are often ineffective, ignored, or misinterpreted. The authors argue that this inefficacy stems not merely from the quality of individual techniques, but from the structural design of conventional XAI interfaces.

In current systems, users are required to select a specific explanation technique before receiving any output. This creates a mismatch between the user's natural-language expression of uncertainty (their "epistemic state") and the discrete technical controls offered by the interface. The paper formalizes this as a failure to bridge the gap between a user's query and the specific explanation technique capable of resolving that query.

Methodology: Formalization and Logical Argumentation
The authors employ a philosophical and logical framework to formalize the selection problem, proceeding through a premise-conclusion structure:

Epistemic Mismatch (Premise 1): AI models are functions mapping inputs to outputs. Explanation techniques are functions mapping inputs and models to specific explanation types. Crucially, different techniques answer structurally different questions. A user's uncertainty is defined as an epistemic state $\omega = (x, y, \phi)$ , where $\phi$ is the specific gap in understanding. Only a subset of techniques, denoted $E^*(\omega)$ , can resolve a specific $\omega$ .
Translation Burden (Premise 2): Users express uncertainty in natural language ( $q \in L$ ), not as technical selections. However, conventional interfaces require a discrete technique selection ( $\hat{e} \in E$ ) as input. Consequently, the user must perform a translation function $f_{user}: L \to E$ .
Unreliability of Mapping (Premise 3): The paper posits that reliable mapping from natural-language questions to XAI technique types is not a skill the general user population possesses. Empirical evidence (e.g., difficulties in using the XAI Question Bank) suggests users often cannot distinguish between technique capabilities.
The Selection Problem: Because users cannot reliably map their uncertainty to the correct technique, the selected technique $\hat{e}$ frequently falls outside the adequate set $E^*(\omega)$ . Formally, the selection problem occurs when $\hat{e} \notin E^*(\omega)$ .

Proposed Structural Solution
To address this, the authors propose shifting the mapping burden from the user to the system. They outline three requirements for a structural solution:

Natural Language Input: The system must accept natural-language queries directly, bypassing the need for users to select technical modes.
Adequate Routing: The system must output techniques that belong to the adequate set $E^*(\omega)$ for the user's specific uncertainty.
System-Performed Translation: The mapping function ( $f_{sys}: L \to E^*(\omega)$ ) must be executed by the system, potentially allowing for multi-turn dialogue to refine the user's intent before routing.

Instantiation: Multi-Agent LLM Orchestration
The paper provides a conceptual instantiation of this solution using a Multi-Agent LLM Orchestration System, specifically in the context of medical imaging.

Architecture: The system utilizes a "Smart" mode where a central LLM orchestrator (e.g., Gemini 2.5 Flash) receives the user's natural-language query.
Mechanism: The orchestrator classifies the underlying type of the user's question and uses function calling to route the query to a specialized XAI agent (e.g., one agent for counterfactuals, another for saliency maps).
Flexibility: Users who already know their desired technique can bypass the orchestrator and invoke specific "modes" directly.
Goal: This design aims to absorb the mapping function into the system, ensuring the explanation technique matches the structural nature of the user's uncertainty.

Key Contributions

Formalization of the Selection Problem: The paper defines the selection problem as a structural failure in conventional XAI interface design where the user is forced to map natural-language uncertainty to technical techniques without adequate support.
Logical Framework: It provides a rigorous logical derivation (Premise 1–3) demonstrating why user-performed mapping is inherently unreliable and leads to functional irrelevance of the explanation.
Design Requirements: It establishes three specific requirements for future XAI interfaces: natural-language input, adequate technique routing, and system-side translation.
Prototype Concept: It offers a concrete example of how a multi-agent LLM orchestration tool can instantiate these requirements, moving XAI from static outputs to interactive, query-driven interactions.

Significance and Claims
The authors modestly claim that their work addresses the "adoption gap" in XAI by reframing it as an interface presentation problem rather than solely a technical algorithm problem. They argue that the inability of users to select the right tool is a structural consequence of current design, not a user deficiency.

The paper does not claim to have empirically validated the proposed multi-agent system with user studies. Instead, it presents the formalization and the architectural proposal as a necessary step for the human-centered XAI community. The significance lies in shifting the design paradigm: future XAI systems should not just offer more techniques, but should actively manage the selection process to ensure the explanation provided actually resolves the user's specific uncertainty. Future work is suggested to empirically test the reliability of user mapping and compare the effectiveness of orchestrators against traditional static interfaces.

Technical Summary: Addressing the Selection Problem in Explainable AI

More like this