Actionable Interpretability Must Be Defined in Terms of Symmetries

Technical Summary: Actionable Interpretability Must Be Defined in Terms of Symmetries

1. Problem Statement

The paper argues that current research in interpretable Artificial Intelligence (AI) is fundamentally ill-posed. While numerous definitions of interpretability exist (e.g., Kim et al., 2016; Miller, 2019; Doshi-Velez & Kim, 2017), they are largely informal and descriptive. Consequently, the field lacks:

1. A formal framework to verify whether a model is interpretable.
2. A set of design principles to construct interpretable models.

Existing definitions fail to specify how interpretability can be formally tested or engineered. The authors posit that without a rigorous foundation, key limitations are overlooked, and the field remains fragmented into lists of desirable properties (e.g., sparsity, modularity) rather than unified principles.

2. Methodology: The Symmetry-Based Framework

Inspired by the Erlangen Program (Klein, 1893), the authors propose that an actionable definition of interpretability must be formulated in terms of symmetries (structure-preserving transformations). They adopt a compositional probabilistic view using Markov categories (specifically the category BorelStoch) to formalize human users and models.

The methodology introduces four specific symmetries that, when satisfied, characterize the class of models interpretable to a given human user $h$:

• Symmetry I: Inference Equivariance

  • Definition: A model $P_{Y|X}$ is interpretable if a user can "simulate" the model's decision process. Formally, there must exist a "mental model" $P^{[h]}_{Y|X}$ in the user's hypothesis space $H_m$ such that translating inputs to the user's space and predicting yields the same result as predicting in the original space and translating the output.
  • Mechanism: This is represented by a commutative diagram where the path of translation-then-prediction equals prediction-then-translation.
  • Limitation: Naively verifying this is intractable for large input spaces (Challenge C1).

• Symmetry II: Information Invariance

  • Definition: To address intractability, the model must retain only input information sufficient for the task, discarding irrelevant details.
  • Mechanism: A surjective map $P_{Z|X}$ compresses the input $X$ into a latent variable $Z$ such that the mutual information $I(Y; X)$ is invariant under marginalization ($I(Y; X|Z) = 0$).
  • Result: This reduces the verification of inference equivariance from the exponential space of $X$ to the tractable space of $Z$.

• Symmetry III: Concept-Closure Invariance

  • Definition: To ensure the translation between model outputs and human understanding is "sound" (preserves meaning), the model must operate on concepts that align with human semantics.
  • Mechanism: Using Formal Concept Analysis, a "concept" is defined as a fixed point of a relation between objects and sentences. A translation $\tau_C$ is sound if it preserves concept closure (the diagram of objects, sentences, and their mappings commutes).
  • Result: This ensures that the latent variables used by the model correspond to human-understandable concepts, preventing conflicting interpretations.

• Symmetry IV: Structural Invariance

  • Definition: The model's internal structure must be compatible with the user's cognitive capabilities (their "mental model" hypothesis space $H_m$).
  • Mechanism: If a user can only reason linearly, the model must be linear. Formally, structural properties are preserved under a functor $F$ mapping the category of interpretable models to the category of human mental models.
  • Result: This makes interpretability explicitly user-centric; a model is only interpretable if its structural complexity matches the user's reasoning capacity.

3. Key Contributions

A. Formal Definition of Interpretable Models
The authors define an interpretable model as one satisfying all four symmetries. They construct a Category of Interpretable Models ($I_m$) as a subcategory of BorelStoch.

• Objects: Concept spaces ($C_1, \dots, C_k$).
• Processes: Concept-based conditional probabilities, copy maps, and discard maps.
• Composition: Sequential and parallel composition rules.
  This category serves as a "recipe" (via string diagrams) for building any interpretable model.

B. Unification of Inference Types as Bayesian Inversion
The paper demonstrates that under this framework, three critical types of inference used in interpretability are unified as forms of Bayesian inversion:

1. Concept Alignment: Learning human concepts by computing the posterior distribution of parameters given ground-truth concept evidence.
2. Interventions: Updating the model's state based on external interventions (e.g., $do$-calculus or ground-truth overrides) via posterior inference.
3. Counterfactuals: Rewriting the model to isolate exogenous variables (abduction), intervening, and predicting outcomes in the modified model.

C. Derivation of Existing Properties
The framework shows that properties often cited as desirable in the literature (sparsity, compactness, faithfulness, modularity) are not primitive axioms but consequences of satisfying the proposed symmetries. For instance:

• Sparsity and compactness arise from Information Invariance.
• Faithfulness and alignment arise from Concept-Closure Invariance.
• Linearity or monotonicity requirements arise from Structural Invariance.

4. Results and Claims

The paper does not present new experimental benchmarks or empirical results. Instead, its "results" are theoretical and structural:

• Tractability: By enforcing Information Invariance, the paper claims to solve the intractability of verifying inference equivariance in high-dimensional spaces.
• Soundness: By enforcing Concept-Closure Invariance, the paper provides a formal criterion to distinguish sound translations (e.g., "black" $\to$ "noir") from unsound ones.
• User-Centricity: By enforcing Structural Invariance, the paper formalizes that interpretability is not an intrinsic property of a model but a relationship between the model and a specific user's hypothesis space.
• Unification: The paper claims to unify disparate inference tasks (alignment, intervention, counterfactuals) under a single mathematical operation (Bayesian inversion within a Markov category).

5. Significance and Call to Action

The paper argues that its significance lies in shifting the field from informal enumeration of properties to formal verification via symmetries.

• Rebuttal to Skepticism: The authors address the view that "no universal mathematical definition exists" by arguing that symmetries provide a universal framework that incorporates user-centrism (via $H_m$) without sacrificing mathematical rigor.
• Rebuttal to Accessibility Concerns: They argue that while the formalism (category theory) is rigorous, the symmetries admit intuitive, example-level interpretations accessible to a broad audience.
• Future Direction: The authors call on the community to:
  1. Map existing interpretability methods to these four symmetries.
  2. Identify which symmetries are implicitly assumed or violated in current work (e.g., noting that Concept Bottleneck Models often satisfy Concept-Closure but may fail Structural Invariance if they use unconstrained DNNs).
  3. Develop new methods that are explicitly "symmetry-complete" for their intended tasks and users.

In summary, the paper posits that actionable interpretability is not a vague quality but a specific structural property defined by how a model preserves information, concepts, and structure relative to a human user, all of which can be formally verified and designed using the proposed symmetry framework.