Interpreto: An Explainability Library for Transformers

1. Problem Statement

Transformer-based language models (from early BERT variants to modern LLMs) are widely deployed for classification and text generation, yet their internal decision-making processes remain opaque. This lack of transparency hinders critical tasks such as debugging, bias detection, and safety assurance.

While numerous libraries exist for model interpretability, they suffer from significant fragmentation:

• Split Capabilities: Tools are often specialized for either attribution methods (feature importance) or mechanistic interpretability (concept-based analysis), rarely offering both.
• Task Limitations: Many tools support only classification or only text generation, not both.
• Pipeline Complexity: Concept-based analysis typically requires stitching together multiple disparate libraries for model splitting, activation extraction, concept learning, labeling, and scoring.
• Accessibility: Documentation, standardized metrics, and end-to-end workflows are often missing, making it difficult for practitioners to adopt research-grade methods.

2. Methodology

INTERPRETO is an open-source Python library designed to unify these fragmented workflows. It bridges recent research and practical tooling by providing a single, unified API for HuggingFace models. The library is built on two complementary pillars:

A. The Attribution Module

This module focuses on feature-level attributions, estimating the contribution of input elements (tokens, words, or sentences) to a model's prediction.

• Supported Tasks: Sequence classification and text generation (CausalLM).
• Methodologies: It implements 10 distinct attribution methods categorized into:
  • Perturbation-based: KernelSHAP, LIME, Occlusion, Sobol.
  • Gradient-based: GradientSHAP, Integrated Gradients, Saliency, SmoothGrad, SquareGrad, VarGrad.
• Features:
  • Granularity Control: Explanations can be generated at the token, word, or sentence level (defaulting to word-level to avoid subword confusion).
  • Target Specification: Users can specify which class (for classification) or which output token (for generation) to explain.
  • Faithfulness Metrics: Includes Insertion and Deletion metrics to evaluate explanation quality.

B. The Concepts Module

This module focuses on concept-based explanations (a component of mechanistic interpretability), identifying higher-level features learned by the model from its activations. It provides a unique end-to-end pipeline that automates the entire workflow:

1. Model Splitting: Uses nnsight to split the HuggingFace model into a feature extractor and a predictor, allowing activation extraction at specific layers.
2. Concept Learning: Trains a concept model on extracted activations to discover recurring patterns. It supports 15 learning options, including:
   • Dictionary Learning: KMeans, PCA, SVD, ICA, NMF, Semi-NMF, Convex NMF.
   • Sparse Autoencoders (SAEs): Vanilla, Jump-ReLU, Top-k, Batch Top-k, Matching Pursuit.
   • Neuron-based: Treating individual neurons as concepts.
3. Interpretation: Assigns human-readable labels to concept dimensions using:
   • Top-k vocabulary tokens.
   • Top-k activating examples (MaxAct).
   • LLM-based labeling: Using external LLMs to generate descriptive labels for concepts.
4. Scoring: Estimates the importance of concepts relative to the model output using concept-to-output gradients.

3. Key Contributions

• Unified API: INTERPRETO is the first library to offer a single interface for both attribution and concept-based explanations across both classification and text generation tasks.
• End-to-End Concept Pipeline: It uniquely integrates model splitting, activation extraction, concept learning, interpretation, and scoring into a single package, significantly reducing the engineering overhead for researchers and practitioners.
• Extensibility: The library is designed with a modular architecture. Users can easily implement custom methods by inheriting from base classes (e.g., Perturbator, Aggregator, ConceptModel) and focusing only on the computational core.
• Comprehensive Benchmarking & Validation:
  • Supports over 15 model architectures (BERT, RoBERTa, GPT-2, Llama 3, Qwen, etc.).
  • Includes unit tests, sanity checks (e.g., verifying attributions outperform random baselines), and integration with industrial use cases (e.g., aviation safety).
• Open Ecosystem: Released under the MIT license with a dedicated documentation site, interactive tutorials, and a demonstration website featuring precomputed explanations for various models and datasets.

4. Results and Evaluation

• Functionality: The library successfully demonstrated the ability to explain models like DistilBERT (emotion recognition), BERT (AG-News classification), and Qwen3-0.6B (text generation).
• Performance:
  • Attribution: Typically requires 10–100 forward passes or 5–20 gradient computations, taking seconds per sample.
  • Concept Pipelines: Dominated by activation extraction and scoring. Small runs (e.g., Semi-NMF on 100 samples) take minutes on an RTX 3080, while large SAE training can take hours.
• Case Study (User Scenario): The paper illustrates a debugging scenario where a practitioner uses INTERPRETO to diagnose an emotion recognition model. By switching from token-level attributions to class-wise concept analysis, they identified that "fear" and "sadness" concepts were not well-separated and that inconsistent class-index mapping across data sources was causing errors.
• Adoption: The library has already been used in an industrial setting (Thales Avionics) for critical sector explainability, validating its practical utility.

5. Significance

INTERPRETO represents a significant step forward in democratizing advanced interpretability techniques. By abstracting away the complex engineering required to stitch together model splitting, concept learning, and labeling, it lowers the barrier to entry for:

• Practitioners: Enabling easier debugging and bias detection in production models.
• Researchers: Providing a standardized framework for benchmarking new interpretability methods.
• Safety Auditors: Facilitating the identification of failure modes and recurring biases in LLMs.

The library's focus on mechanistic interpretability (via concepts) alongside traditional attributions positions it as a comprehensive tool for understanding how models work, rather than just what they predict. Future work plans include adding supervised concept options (probes, CAVs), expanding to multi-modality, and tighter integration between attribution and concept analyses.