🔬 physics

Interpreting "Interpretability" and Explaining "Explainability" in Machine Learning in Physics

This paper reviews the concepts of interpretability and explainability in machine learning for physics, defining them as structural transparency and scientific content respectively, while arguing that they are deliberate modeling choices subject to trade-offs rather than inherent properties of the models themselves.

Original authors: Rikab Gambhir, Luisa Lucie-Smith, Jesse Thaler

Published 2026-06-26

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Rikab Gambhir, Luisa Lucie-Smith, Jesse Thaler

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Big Picture: Two Different Questions

Imagine you are handed a mysterious, high-tech machine that predicts the weather. You want to know two things about it:

How does the machine work? (What are the gears, wires, and levers inside?)
What does the machine actually tell us about the world? (Does its prediction match what we already know about meteorology?)

The authors of this paper argue that scientists often confuse these two questions. They propose a clear distinction:

Interpretability is about the structure. It asks: "Can I see inside the machine and understand how the gears turn to make a prediction?"
Explainability is about the content. It asks: "Does the machine's answer make sense when I compare it to the map of the world I already have?"

The Trade-Offs: The "Swiss Army Knife" vs. The "Screwdriver"

The paper explains that you usually can't have the best of both worlds at the same time. You have to choose your tool based on the job.

1. Interpretability vs. Expressivity (The "Simple Tool" vs. The "Power Tool")

The Analogy: Think of a simple screwdriver (Interpretable) vs. a complex, programmable Swiss Army knife with a thousand attachments (Expressive).
The Trade-off: A screwdriver is easy to understand; you know exactly what it does. But it can only tighten screws. A Swiss Army knife can do almost anything (cut, screw, saw, open bottles), but if you look at the inside, it's a tangled mess of gears and springs that is very hard to figure out.
The Paper's Point: In physics, simple models (like Newton's laws) are like screwdrivers: easy to understand but limited. Complex Machine Learning (ML) models are like the Swiss Army knife: they can solve incredibly hard problems, but they are often "black boxes" where we can't see how they work.

2. Explainability vs. Adaptability (The "Rigid Map" vs. The "Flexible GPS")

The Analogy: Imagine a printed map of a city (Explainable) vs. a GPS that learns from every driver's route in real-time (Adaptable).
The Trade-off: The printed map is based on known roads and rules. If you see a new road on the map, it's a mistake. It's very "explainable" because it follows the rules of the city. The GPS, however, can adapt to traffic jams, construction, and detours instantly. But because it changes so much based on data, it's hard to explain why it chose a specific route at a specific moment.
The Paper's Point: If you want a model that fits any data perfectly (Adaptability), you lose the ability to explain it using known physics. If you want a model that strictly follows known physics (Explainability), it might fail to fit new, weird data.

When Do You Need Which One?

The paper says you don't always need both. It depends on what you are trying to do.

When you don't need Interpretability:
- The "Surrogate" Job: Imagine you have a super-slow, complex simulation that takes a week to run. You train a fast AI to mimic it. You don't care how the AI does it (Interpretability); you just care that it gives the same answer as the slow simulation and runs in a second.
- The "Benchmark" Job: If you are just trying to win a competition to see who has the highest score, you might use a "black box" model. As long as it wins, you don't need to know how it thinks.
When you DO need Interpretability:
- Trust: If a model makes a weird prediction, you need to be able to look inside and say, "Ah, I see, it got confused because of this specific input."
- Efficiency: Sometimes, understanding the "gears" allows you to build a simpler, faster version of the machine.
When you DO need Explainability:
- Discovery: If your model finds something new, you must be able to say, "This new pattern matches the theory of X." If you can't connect the result to existing knowledge, it's just a list of numbers, not a discovery.
- Validation: You need to prove the model isn't just memorizing the data (like a student memorizing answers) but actually understanding the rules of the universe.

The "By-Design" vs. "After-the-Fact" Approach

The paper discusses two ways to get these qualities:

Intrinsic (By-Design): You build the machine with the "gears" visible from the start.
- Analogy: Building a house with glass walls so you can see the plumbing.
- Example: Using a simple equation or a model that forces the AI to follow the laws of physics (like conservation of energy) while it learns.
Post-Hoc (After-the-Fact): You build a complex machine first, then try to take a picture of the inside to see how it works.
- Analogy: Taking a complex engine apart after it's built to draw a diagram of how the pistons move.
- Example: Using tools that highlight which parts of an image the AI looked at to make a decision. The paper warns that these are often just approximations and might not tell the whole truth.

The Most Important Lesson: Plan Before You Build

The authors emphasize that Interpretability and Explainability are choices, not magic properties.

Task Specification: Before you build your model, you must ask: "What is this model actually for?" If you are just trying to speed up a calculation, you don't need a transparent model. If you are trying to discover a new law of physics, you absolutely do.
Intervention Planning: You must decide beforehand what you will do if the model is confusing.
- Analogy: If you are driving a car, you need to know what you will do if the engine light comes on. Do you pull over? Do you ignore it? If you don't have a plan, seeing the light is useless.
- The paper warns against "confirmation bias." If you look at a model's inner workings after you see the result, you might trick yourself into thinking it makes sense just because you want it to. You need a plan to test the model before you look at the results.

Conclusion

The paper concludes that Machine Learning models in physics are not fundamentally different from the "classical" models scientists have used for centuries (like Newton's laws). The only difference is scale.

Classical models are just very simple, highly interpretable, and highly explainable.
ML models are huge, complex, and often opaque.

But the rules of science apply to both. You can't just throw a "black box" at a scientific problem and expect a discovery. You have to decide what you need (speed? trust? discovery?), choose the right tool, and have a plan for how to use the results. If you do that, ML becomes a powerful instrument for science, not just a magic trick.

Technical Summary: Interpreting "Interpretability" and Explaining "Explainability" in Machine Learning in Physics

Problem Statement

The rapid adoption of highly expressive machine learning (ML) models, such as deep neural networks and large language models (LLMs), in physical sciences (high-energy physics, astrophysics, cosmology) has raised fundamental questions regarding their scientific reliability and role in discovery. A primary challenge is the conflation of terms like interpretability, explainability, trustworthiness, and transparency within the physics community. Unlike "classical" scientific models (e.g., Newton's Laws, the Standard Model), which are often derived from first principles and possess inherent structural transparency, modern ML models are frequently viewed as opaque "black boxes."

The paper addresses the need to rigorously define and distinguish between interpretability and explainability to determine when these properties are necessary, how they can be achieved, and what trade-offs they entail. The authors argue that these are not inherent properties of models but deliberate modeling choices that must be aligned with specific scientific goals.

Methodology and Framework

The paper employs a conceptual review and taxonomy framework rather than presenting new experimental data or a specific algorithm. The methodology involves:

Definitional Clarification: Establishing distinct definitions for interpretability and explainability, separating model structure from scientific content.
Taxonomy Construction: Mapping models onto a spectrum based on their degree of interpretability and explainability, identifying four quadrants: interpretable/explainable, interpretable/not explainable, not interpretable/explainable, and neither.
Trade-off Analysis: Analyzing the inherent tensions between:
- Interpretability vs. Expressivity: The ability to understand a model's structure versus its capacity to fit complex data patterns.
- Explainability vs. Adaptability: The ability to map a model to domain knowledge versus its flexibility to fit arbitrary data distributions.
Categorization of Techniques: Classifying methods for achieving these properties into Intrinsic (by-design) and Post-Hoc (analysis after training) approaches.
Process Integration: Framing interpretability and explainability as prerequisites for Task Specification and Intervention Planning, arguing that these properties are only valuable if they inform downstream scientific decisions.

Key Contributions

1. Distinct Definitions

The authors propose a clear dichotomy:

Interpretability: Concerns the structural transparency of a model. It asks: "Can we understand, approximate, or read off the inner workings of the model $f_\theta$ and how it reaches its output?" It is data-agnostic and relates to the model's form, parameters, and computational mechanism (e.g., linear coefficients, decision tree splits).
Explainability: Concerns the scientific content of a model. It asks: "Can we map the learned parameters or functions onto existing domain knowledge?" It requires a reference to prior physical principles, laws, or empirical data to provide context and semantics.

2. The Landscape of Models

The paper categorizes models based on these definitions:

Interpretable & Explainable: Classical physics models (Newtonian gravity, Standard Model) and simple ML models (linear regression, shallow trees).
Interpretable but Not Explainable: Models where the structure is clear (e.g., inspecting CNN kernels) but the connection to physical causality or domain knowledge is weak or qualitative.
Explainable but Not Interpretable: Models where the scientific outcome is understood (e.g., a QCD prediction at 10-loops or a normalizing flow mimicking it), but the internal computational mechanism is too complex to be mechanistically transparent.
Neither: Proprietary black boxes or models with no access to structure or domain mapping.

3. Trade-offs

Interpretability vs. Expressivity: Highly expressive models (e.g., Transformers, Deep Neural Networks) can approximate vast classes of functions but sacrifice structural transparency. Conversely, highly interpretable models (e.g., parametric fits) are often limited in their ability to capture complex, non-linear data patterns.
Explainability vs. Adaptability: Models designed for high explainability (incorporating physical symmetries, priors, or first principles) often have reduced adaptability to arbitrary or unexpected data distributions. Conversely, highly adaptable models (e.g., look-up tables) can fit any data but offer no compressible structure to explain why the data behaves as it does.

4. Methodological Tools

The paper reviews tools for achieving these properties:

Intrinsic (By-Design): Physics-informed architectures (e.g., Lorentz-invariant networks), representation learning (e.g., disentangled latent spaces, SciNet), symbolic regression, and sparse autoencoders.
Post-Hoc: Metric-based tools (Shapley values/SHAP, Saliency maps, LIME) and distillation techniques (training a simpler "student" model to mimic a complex "teacher").

5. Task Specification and Intervention Planning

A central contribution is the argument that interpretability and explainability are only meaningful when tied to Task Specification (defining the scientific question and desired properties) and Intervention Planning (defining what actions will be taken based on the interpretation).

The authors assert that if a model is interpreted but no downstream decision (e.g., changing a hypothesis, adjusting uncertainties, rejecting a model) follows, the effort may be superfluous.
They warn against confirmation bias, emphasizing that intervention plans must be pre-specified to avoid post-hoc rationalization of results.

Results and Claims

The paper does not present numerical results from a specific experiment but offers a structured analysis of the field:

Universality of the Problem: The authors claim that interpretability and explainability are not unique challenges to ML; classical models face the same questions, differing only in scale and complexity.
Necessity of Context: There is no scientific application where both interpretability and explainability are entirely unnecessary. However, the degree required varies by task (e.g., scientific discovery requires both; surrogate modeling for speed may prioritize explainability over interpretability).
Current State of AI in Physics: The paper claims that while ML has significantly improved measurements and discovery potential (e.g., jet tagging at the LHC), the field has not yet widely achieved "learning from" ML in the sense of discovering novel physical laws via explainable AI. Most progress remains in improving sensitivities and bounds rather than revealing hitherto unknown patterns.
Design Philosophy: The authors conclude that interpretability and explainability should be treated as deliberate modeling choices rather than inherent properties. They advocate for designing models, loss functions, and training techniques that align with specific scientific goals, rather than treating these properties as an afterthought.

Significance

The paper positions itself as a foundational guide for the physics community (specifically within the VERaiPHY initiative) to standardize the discourse around ML model validation. Its significance lies in:

Demystifying Terminology: Providing a rigorous, operational distinction between interpretability (structure) and explainability (content), preventing the conflation of terms that hinders scientific communication.
Shifting the Paradigm: Moving the conversation from "how to make black boxes transparent" to "what modeling choices are required to achieve specific scientific goals."
Establishing Standards: Arguing that for ML to serve as a genuine instrument of scientific knowledge, models must be subject to the same questions of validity, trust, and domain alignment as classical models.
Practical Guidance: Offering a framework for researchers to decide when to invest in interpretability/explainability and how to integrate these considerations into the scientific workflow via task specification and intervention planning.

The authors emphasize that while tools for post-hoc analysis exist, the most robust path forward involves intrinsic design choices that embed physical knowledge directly into the model architecture, ensuring that the resulting models are not just predictive, but scientifically grounded.