Identifying Confusion Trends in Concept-based XAI for Multi-Label Classification
This paper evaluates Concept-based Explainable AI (CXAI) methods, specifically CRP and CRAFT, on deep neural networks trained for multi-label classification using the MS-COCO dataset, revealing that these techniques effectively identify model learning weaknesses, demonstrate how concept distinctiveness reduces confusion, and expose dataset-induced biases.
Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer
In the world of modern artificial intelligence, computers have become remarkably good at looking at pictures and naming what they see. They can identify a dog, a car, or a person with a level of accuracy that often surpasses human vision. Yet, for these systems to be trusted in high-stakes situations like self-driving cars or medical diagnosis, we need more than just a correct answer; we need to understand how the machine arrived at that conclusion. This is the realm of explainable AI, a field dedicated to opening the black box of deep neural networks. Within this field, a specific approach called concept-based explainability has emerged. Instead of simply highlighting the pixels that mattered most, this method tries to identify the meaningful ideas or "concepts" the computer has learned to recognize. It asks not just where the computer is looking, but what it thinks it is seeing. The challenge arises when these systems operate in complex, real-world environments where objects overlap, backgrounds are cluttered, and a single image contains many different things. In such scenarios, even highly accurate models can make subtle mistakes, confusing one object for another or relying on misleading background clues. Understanding why these confusions happen is critical for building safer, more reliable technology.
Researchers at the Technical University of Dresden set out to investigate these confusion patterns by training two well-known computer vision models, ResNet50 and VGG-16, on a massive collection of images known as MS-COCO. This dataset contains over one hundred thousand photographs of everyday scenes, each tagged with multiple labels to describe the various objects present. The team focused their study on the twenty most frequently appearing objects in these images, such as people, cars, and chairs. To ensure a fair test, they trained the models to different levels of performance, creating scenarios where the computers were either highly accurate or struggling significantly. They then applied two distinct methods for generating explanations, one that traces the importance of specific features and another that breaks images down into smaller, focused regions. By comparing the computer's final predictions with the internal concepts it used to make those decisions, the team could see exactly where the models were getting confused.
The investigation revealed that when a model struggles to distinguish between two similar objects, the confusion is not random; it is rooted in the specific concepts the computer has learned. In cases where the models performed well, the internal concepts were sharp and distinct, clearly separating a person from a car or a chair. However, when the models performed poorly, the concepts became muddy and overlapping. The computer began to rely on shared features that were not unique to a single object, leading to errors. For instance, the study found that a model might confuse a person with a backpack not because it cannot see the person, but because it has learned to associate the concept of "person" heavily with the visual patterns of a backpack, simply because they appear together so often in the training data. This suggests that the confusion stems from the model learning spurious connections rather than the true identity of the object.
A key finding of the research is that the clarity of these internal concepts directly correlates with the model's ability to avoid mistakes. When the concepts the computer learned were distinct and unique to their specific class, the rate of confusion dropped significantly. Conversely, when the concepts were vague or shared too much information with other classes, the model made more errors. The researchers measured this "distinctiveness" and found that models with higher distinctiveness scores had fewer instances of misidentifying objects. This indicates that the path to better artificial intelligence lies not just in getting the right answer more often, but in ensuring the computer learns features that are truly unique to each object, rather than relying on common background patterns.
The study also uncovered how the environment itself can trick the computer. In many real-world images, certain objects appear together so frequently that the computer starts to believe they are part of the same thing. For example, in a dataset of train stations, the computer might learn to recognize a "person" by looking for the platform or the staircase, because people are almost always standing on those structures in the training photos. When the researchers tested the model on new images, it would identify a person even if the person was standing on a different surface, simply because the background matched what it had learned. These "environmental concepts" act as hidden biases, where the model relies on the setting rather than the object itself. This was particularly evident in a dataset for autonomous train driving, where the model's explanations showed it was fixated on the background tracks and platforms rather than the people themselves.
Ultimately, the work demonstrates that concept-based explanations are powerful tools for diagnosing why a model fails. By looking at the concepts a computer uses, researchers can see if a model is learning the right things or if it is being led astray by the way the data was collected. The findings suggest that to build more robust systems, we must be careful about the data we feed them, ensuring it is diverse enough to prevent these misleading associations. If a model learns to recognize a person only when they are near a car, it will fail when that person is in a park. The study confirms that improving the distinctiveness of learned concepts and reducing reliance on environmental cues are essential steps toward creating artificial intelligence that can be trusted in the complex, unpredictable real world.
Drowning in papers in your field?
Get daily digests of the most novel papers matching your research keywords — with technical summaries, in your language.