Technical Summary: In Defense of Information Leakage in Concept-based Models

Problem Statement

Concept-based models (CMs) are deep neural networks designed to ground predictions in human-understandable concepts (e.g., "stripes," "wings"). A prevailing narrative in the field posits that information leakage—where concept representations encode information not strictly attributable to their intended semantics—is inherently undesirable. This leakage is typically categorized into inter-concept leakage (a concept representation encodes information about other concepts) and task leakage (a concept representation encodes information about the downstream task label $Y$).

The conventional view argues that leakage undermines intervenability (the ability to correct predictions by modifying concept inputs) and interpretability (the ability to reconstruct model reasoning solely from concepts). Consequently, recent research has focused on eliminating leakage through architectural constraints or regularization.

However, the authors argue that this strict view is ill-posed and counterproductive, particularly in real-world settings where concept incompleteness is the norm. In these scenarios, the set of available training concepts $C$ often fails to fully describe the downstream task $Y$ (i.e., $I(Y; X | C) \neq 0$). The authors posit that enforcing strictly non-leaky representations in such incomplete settings precludes high task accuracy, rendering CMs impractical.

Methodology

1. Formalizing Benign Leakage

The paper introduces the concept of Benign Leakage, defined as a specific form of leakage that is necessary and desirable for accurate CMs in incomplete settings. A representation $\hat{C}$ exhibits benign leakage if it satisfies two conditions:

• Sufficiency: The representation preserves all label-relevant information not captured by the ground-truth concepts $C$. Formally, $I(Y; R | C) \approx I(Y; X | C)$, where $R$ is a residual component of the representation.
• Localization: The information about a specific concept $C_i$ that is useful for predicting $Y$ is localized within a specific component $\bar{C}_i$ of the representation. Formally, $I(Y; \bar{C}_i | R_i, \hat{C}_{-i}) \approx I(Y; C_i | C_{-i})$.

This decomposition allows a model to exploit leakage for accuracy (via $R$) while maintaining intervenability (via $\bar{C}_i$).

2. Optimization via $L_{int}$

To achieve benign leakage, the authors propose optimizing a new training objective, $L_{int}$ (Intervention Loss). This loss minimizes the task loss when all concepts are intervened upon (i.e., forced to their ground-truth values during training):
$$L_{int} = \mathbb{E}_{(x,c,y) \sim D} \left[ -\log P_f(y | \hat{C} = g(x | C := c)) \right]$$

The paper argues that minimizing $L_{int}$ implicitly encourages sufficiency. Under the assumption that the label predictor is well-specified, minimizing the negative log-likelihood under full intervention is equivalent to maximizing the mutual information between the residual representation and the task label, given the concepts.

Furthermore, the authors hypothesize that due to the simplicity bias of Deep Neural Networks (SGD favors simpler hypotheses), minimizing $L_{int}$ also implicitly encourages localization. A simpler hypothesis for the model is one where the label predictor learns to rely on the concept-aligned components for concept-specific reasoning, rather than re-encoding concept information into the residual leakage.

3. Experimental Setup

The authors evaluate their claims using the CUB-200 dataset (birds), specifically constructing an incomplete version (CUB-Incomplete) by subsampling concept annotations to 22 concepts (7 groups) out of the original 112. They test various architectures, including:

• Independent/Sequential/Joint CBMs: Standard baselines with varying degrees of leakage.
• Hybrid CBMs: Models with bypass channels (residual connections) that explicitly allow leakage.
• Concept Embedding Models (CEMs) & ProbCBMs: Models using dynamic embeddings.
• Black Box Baselines: DNNs with no concept alignment loss.

Key metrics include Task Fidelity (accuracy), Concept Fidelity (accuracy of concept predictions), and Intervenability (performance improvement as ground-truth concepts are provided at test time).

Key Results

1. Leakage Does Not Necessarily Destroy Intervenability

Contrary to the prevailing narrative, the authors demonstrate that leaky models can remain highly intervenable in incomplete settings, provided they are properly conditioned.

• Without $L_{int}$: Many leaky models (e.g., Joint Logit CBMs, Hybrid CBMs with large bypasses) become unintervenable in incomplete settings; their accuracy does not improve (or decreases) when concepts are corrected.
• With $L_{int}$: When these same leaky models are trained with the $L_{int}$ regularizer, they regain high intervenability. For instance, large Hybrid CBMs (with 500 residual neurons) become as intervenable as non-leaky Independent CBMs, while retaining significantly higher task accuracy.

2. Benign Leakage Improves Accuracy Without Sacrificing Interpretability

• Accuracy: In incomplete settings, models trained with $L_{int}$ achieve significantly higher task fidelity than non-leaky baselines (e.g., Independent CBMs), which are forced to rely solely on the incomplete concept set.
• Interpretability: Using proxy metrics (concept ROC-AUC and label predictor weight analysis), the authors show that well-conditioned leaky models (like Hybrid CBMs with $L_{int}$) exhibit concept-to-task weight structures statistically indistinguishable from non-leaky models. The top concepts used by leaky models to predict classes align closely with those used by non-leaky models, suggesting that leakage does not obscure the model's reasoning process in a way that standard metrics detect.

3. The Role of Incompleteness

Experiments on complete datasets show that leaky models are often intervenable even without $L_{int}$. However, in incomplete settings, the absence of $L_{int}$ leads to unintervenable behavior in leaky models. This confirms that the "harm" of leakage is context-dependent and arises specifically when the model fails to learn a sufficient representation of the missing task-relevant information.

Significance and Claims

The paper claims to shift the paradigm regarding information leakage in CMs from a phenomenon to be eradicated to one to be controlled and exploited.

1. Reframing the Narrative: The authors argue that the strict view of leakage as "uninterpretable" is ill-posed because it ignores the practical necessity of leakage in incomplete real-world scenarios. They contend that eliminating leakage in such settings forces a trade-off where accuracy is sacrificed for a theoretical purity that offers no practical benefit.
2. Practical Necessity: The paper posits that benign leakage is necessary for constructing CMs that are both accurate and intervenable when concept sets are incomplete. Without allowing the model to capture $I(Y; X | C)$ via leakage, CMs cannot perform their primary function of accurate prediction.
3. Methodological Contribution: The introduction of $L_{int}$ provides a simple, architecture-agnostic regularizer that enables "benign leakage." It recovers various existing mechanisms (like stochastic intervention in CEMs) as special cases and offers a unified approach to training robust CMs.
4. Call to Action: The authors call for the community to stop broad claims that leakage destroys interpretability without falsifiable criteria. Instead, they urge researchers to:
   • Evaluate CMs in incomplete settings.
   • Focus on controlling leakage (localization and sufficiency) rather than eliminating it.
   • Separate CM-specific phenomena from general deep learning failure modes like shortcut learning.

In conclusion, the paper asserts that "leakage" is not inherently malign; rather, uncontrolled leakage is the issue. By optimizing for sufficiency and localization via $L_{int}$, it is possible to build CMs that leverage leakage to achieve high accuracy in incomplete settings while retaining the core properties of interpretability and intervenability.