Technical Summary: OCCAM – Open-set Causal Concept explAnation and Ontology induction for black-box vision Models

Problem Statement

Deep image classifiers have achieved near-human performance, yet their decision-making processes remain opaque, particularly in black-box settings where internal gradients, activations, and architectural details are inaccessible (e.g., deployed systems and third-party APIs). Existing explanation methods face two primary limitations in this regime:

1. Attribution-based methods (e.g., saliency maps, LIME, SHAP) operate on spatial units (pixels/patches) rather than semantic concepts, producing continuous relevance maps that lack explicit semantic structure and require manual interpretation.
2. Concept-based methods typically rely on closed-set predefined vocabularies or curated concept banks, limiting their ability to explain novel or fine-grained entities found in real-world open-set scenarios. Furthermore, most techniques operate locally, explaining individual predictions without providing a principled mechanism to derive consistent global insights or uncover systematic model biases.

There is a critical need for explanation frameworks that are simultaneously black-box, open-set, and causal.

Methodology

OCCAM (Open-set Causal Concept explAnation and ontology induction for black-box vision Models) reframes model interpretation as a process of structured interventional pruning. The framework operates strictly under query-only access, requiring no internal model weights. It consists of three main stages:

1. Local Interventional Concept Analysis

For a given input image $x$ and a pre-trained classifier $f$:

• Open-Set Concept Proposal ($\Phi$): A Multimodal Large Language Model (MLLM) proposes a set of instance-specific semantic concepts $\{k_1, ..., k_N\}$ from an open-ended universe, rather than a fixed vocabulary.
• Concept Grounding ($\Psi$): A text-conditioned segmentation model (e.g., SAM) maps each proposed concept to a binary segmentation mask $m_i$, identifying its spatial support. Concepts failing to produce valid masks are discarded.
• Causal Pruning ($R$): An inpainting model (e.g., LaMa) removes the visual support of each grounded concept independently, creating an intervened image $\tilde{x}_i$.
• Causal Contribution Estimation: The framework measures the change in the classifier's confidence for the originally predicted class $\hat{y}$:
  $$s_{x, k_i} = f_{\hat{y}}(x) - f_{\hat{y}}(\tilde{x}_i)$$
  A larger score indicates that the concept provides stronger positive causal support for the decision.

2. Global Ontology Induction

Beyond local explanations, OCCAM aggregates interventional evidence across the entire dataset to induce a structured conceptual ontology.

• Structure: The ontology is formalized as a knowledge graph (RDF/OWL) where nodes represent classes, images, concepts, and causal evidence entities (containing scores and artifacts).
• Emergence: Global concept–class and inter-concept dependencies are not imposed top-down but emerge from aggregating the stored interventional attributes.
• Reasoning: This ontology enables structured querying and reasoning over the model's global behavior, revealing stable dependencies, compositional patterns, and latent causal relations that isolated per-image explanations might miss.

3. Evaluation Metrics

The framework is evaluated using:

• Causal Pruning Metrics: Average Drop Percentage (ADP), Maximum Drop Percentage (MDP), and Maximum Absolute Logit Drop (MAD) to quantify the impact of concept removal on model confidence.
• Grounding Metrics: Energy-based Pointing Game (EPG), Normalized Region Accuracy (NRA), and Hit Rate to assess spatial alignment with ground-truth masks.
• Reasoning Quality: Reports generated by LLMs using unstructured summaries, flat JSON, and ontology-grounded reasoning are rated by independent LLMs and human evaluators for coherence and faithfulness.

Key Contributions

1. OCCAM Framework: A strictly black-box framework for open-set causal explanation via structured object-level interventions, requiring only query access to the visual classifier.
2. Principled Ontology Construction: A procedure that aggregates concept-level causal effects across images to construct an emergent global conceptual ontology without relying on predefined vocabularies or internal representations.
3. Ontology-Driven Reasoning: Demonstration that reasoning over the induced ontology enables deeper explanations, uncovering stable conceptual dependencies and latent causal relations undetected by isolated per-image analyses.
4. State-of-the-Art Performance: Achievement of competitive or superior results on Broden and ImageNet-S datasets under strict open-set and black-box conditions, outperforming white-box, closed-set competitors in causal pruning effectiveness.
5. Multimodal Extensibility: Validation that the framework extends naturally to vision–language classifiers (e.g., CLIP, SigLIP), providing a unified explanation mechanism across unimodal and multimodal architectures.

Experimental Results

Experiments were conducted on ResNet18, ResNet50, ViT-B/16, and various CLIP/SigLIP variants using the Broden and ImageNet-S datasets.

• Causal Pruning: OCCAM consistently induced stronger confidence reductions than the state-of-the-art closed-set method CE-FAM. For example, on ResNet50, OCCAM achieved an Average Drop Percentage (ADP) of 47.19% compared to CE-FAM's 44.18%, and a Maximum Drop Percentage (MDP) of 93.89% vs. 84.14%. This indicates OCCAM identifies more decision-critical evidence.
• Grounding Validity: Despite operating in a black-box, open-set regime, OCCAM achieved competitive localization scores (e.g., EPG of 0.65 on ResNet50 for ImageNet-S) against white-box methods, demonstrating that object-level interventional grounding produces spatially aligned and semantically meaningful regions.
• Ontology Reasoning: In global explanation evaluations, ontology-grounded reasoning achieved the highest scores (average 4.78/5) compared to unstructured summaries (4.31) and flat JSON (4.23). This suggests that formally structuring causal evidence significantly enhances the coherence and faithfulness of global model explanations.
• Multimodal Robustness: When applied to CLIP and SigLIP, progressive removal of top-3 influential concepts caused a monotonic decrease in zero-shot accuracy. Notably, SigLIP proved more robust than CLIP variants, suggesting its decisions rely on more distributed evidence rather than a small set of high-impact regions.

Significance and Claims

The paper claims that OCCAM offers a paradigm shift in model interpretation by reframing it as structured interventional pruning. Its primary significance lies in:

• Bridging the Gap: It successfully addresses the "black-box, open-set, causal" trilemma, a regime where existing methods struggle.
• Global Insight: By moving beyond per-image attribution, it provides a mechanism to derive consistent global insights and uncover systematic model biases through an emergent ontology.
• Behavior-Grounded Knowledge: The induced ontology is not externally imposed but derived directly from measured causal effects, offering an interrogable representation of how classifiers organize visual evidence.
• Unified Mechanism: It demonstrates that input-level interventions provide a unified explanation mechanism applicable to both traditional vision models and modern multimodal architectures, without requiring access to internal representations.

The authors conclude that this perspective encourages future work on behavior-grounded knowledge induction and structured reasoning for explainable AI.