🤖 AI

LINE: LLM-based Iterative Neuron Explanations for Vision Models

The paper introduces LINE, a training-free, black-box framework that leverages large language models and text-to-image generators to iteratively discover and refine open-vocabulary concept labels for individual neurons in vision models, achieving state-of-the-art performance and uncovering novel concepts missed by predefined vocabularies.

Original authors: Vladimir Zaigrajew, Michał Piechota, Gaspar Sekula, Paweł Gelar, Przemysław Biecek

Published 2026-05-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Vladimir Zaigrajew, Michał Piechota, Gaspar Sekula, Paweł Gelar, Przemysław Biecek

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot that can look at a picture and tell you what it is. But there's a catch: the robot is a "black box." You can see the picture go in and the answer come out, but you have no idea how it decided that answer. Inside the robot's brain are billions of tiny switches called neurons. Some of these switches light up when the robot sees a "dog," others when it sees a "tree," but we don't know which is which.

For a long time, scientists have tried to label these switches, but they've been like detectives trying to guess a suspect's name using only a list of 1,000 pre-written names. If the robot is thinking about something specific, like a "red fire hydrant," but that exact phrase isn't on their list, they might just guess "fire" or "red," missing the real point.

Enter LINE (LLM-based Iterative Neuron Explanations). Think of LINE as a creative detective team that doesn't just guess from a list; it invents the right description through a game of "guess and check."

Here is how LINE works, step-by-step:

1. The Starting Guess (The Scoreboard)

LINE starts with a rough guess. It looks at a list of common things (like "pool table" or "barbell") and sees which ones make the neuron light up a little bit. It puts these on a scoreboard.

Analogy: Imagine you are trying to describe a mysterious object. You start by saying, "Is it a chair? No. Is it a table? Maybe a little."

2. The Creative Loop (The Magic Team)

This is where LINE gets smart. It uses two AI tools working together:

The Writer (LLM): A large language model that acts like a brainstorming partner. It looks at the current scoreboard and says, "Hey, 'pool table' and 'barbell' both light up this neuron. Maybe it's not about the objects themselves, but about exercise?"
The Artist (Text-to-Image Model): Once the Writer suggests a new idea (like "strength training"), the Artist instantly draws 50 different pictures of people lifting weights, doing yoga, or playing sports.

3. The Reality Check (The Test)

LINE takes those 50 new pictures and feeds them back into the robot's brain.

If the neuron lights up brightly for the "strength training" pictures, the Writer gets a high score.
If the neuron stays dark, the score is low.

4. The Refinement (Iterating)

The team repeats this loop.

Round 1: "Maybe it's a gym?" (Score: Okay).
Round 2: "Maybe it's weightlifting?" (Score: Better).
Round 3: "Maybe it's specifically 'strength training'?" (Score: Excellent).

The process keeps going, refining the idea until they find the perfect description. Finally, they take the top few ideas and ask the Writer to summarize them into one big concept (like "physical exercise").

Why is this a big deal?

1. It breaks the "List" rule.
Old methods were stuck with a fixed menu of words. If the robot was thinking about a "four-poster bed," an old method might just say "bed" or "bedroom." LINE realized the robot was actually thinking about the specific style: "four-poster." It found 27% more new concepts that the old lists missed.

2. It draws the picture to prove it.
Instead of just saying a word, LINE draws the image that makes the neuron go crazy. It's like the robot saying, "I know this neuron is for 'spiders' because look at this picture of a spider web I drew—it makes the neuron scream!" This helps prove the label is actually correct.

3. It works without peeking inside.
Most methods require the robot to be "open" so scientists can see the math inside (gradients). LINE works like a black box. It just talks to the robot, shows it pictures, and listens to the reaction. This means it can work on any robot, even the most secret ones.

The Results

The paper tested LINE on several famous robot brains (like ResNet and ViT).

Better Scores: It got significantly higher accuracy scores than previous methods (improving by up to 0.11 points, which is huge in this field).
New Discoveries: It found that many neurons weren't just looking for one thing; they were looking for a category of things (like "exercise" instead of just "barbell").
Visual Proof: The pictures it generated were much clearer and more natural than the weird, blurry images produced by older methods.

The Catch (Limitations)

The authors are honest about the flaws:

Starting Bias: Since LINE starts with a list of common things (like ImageNet), it might miss very rare or specialized things (like specific medical conditions) if they aren't in that starting list.
One Idea at a Time: Sometimes a neuron is "polysemantic," meaning it cares about two totally different things (like "dogs" AND "fire"). LINE usually picks the strongest one and might miss the second one.
The Artist's Mistakes: If the drawing tool (the Text-to-Image model) draws a bad picture, the test might fail, even if the idea was good.

Summary

LINE is a new way to understand AI brains. Instead of forcing the AI to fit into a pre-made box of words, it uses a creative loop of suggesting ideas, drawing them, and testing them to find the exact, human-readable description of what the AI is actually thinking. It's like having a translator that doesn't just guess words, but draws pictures to make sure it got the meaning right.

Technical Summary: LINE (LLM-based Iterative Neuron Explanations for Vision Models)

Problem Statement

Interpreting individual neurons in deep neural networks (DNNs) is essential for understanding complex decision-making processes and ensuring AI safety, particularly in high-stakes domains like healthcare and justice. While mechanistic interpretability aims to uncover localized decision circuits, a significant challenge remains: automatically translating neural activity into human-understandable semantic concepts.

Existing neuron labeling methods suffer from two primary limitations:

Reliance on Predefined Vocabularies: Many approaches (e.g., Network Dissection, CLIP-Dissect) are constrained to fixed concept sets, which often lack the precise ground truth or fail to capture higher-order, global concepts.
Overly Specific or Crisp Descriptions: Methods utilizing text-generative models (e.g., MILAN) often produce descriptions that are too narrow, failing to generalize to broader semantic abstractions.
Gradient Dependence: Activation Maximization (AM) methods, used for visual explanations, typically rely on gradient-based optimization. This imposes memory overhead, requires differentiability, and often limits compatibility to older text-to-image (T2I) models, while producing images with unnatural artifacts.

Methodology: LINE Framework

The authors introduce LINE (LLM-based Iterative Neuron Explanations), a novel, training-free, black-box framework designed for open-vocabulary concept labeling in vision models. LINE operates as a closed-loop iterative process that leverages a Large Language Model (LLM) and a Text-to-Image (T2I) generator without requiring access to the target model's gradients or training data.

The framework consists of four core components operating in a loop:

Initialization: The process begins by initializing a "concept scoreboard" ( $H$ ) using labels from the ImageNet-1K validation set. For each class, a set of images is passed through the target vision model to extract initial neuron activations, establishing a baseline score vector. The scoreboard is populated with the top-scoring initial labels and a set of random labels to balance exploitation and exploration.
Concept Proposal (LLM): In each iteration, an LLM analyzes the current scoreboard (specifically the top-scoring concepts and their history) to propose a new, refined concept description. The LLM is guided to generalize, unify, or creatively combine existing high-scoring concepts while avoiding previously proposed terms.
Image Synthesis (T2I): The proposed concept is passed to a T2I model (e.g., Stable Diffusion XL) to generate a batch of diverse synthetic images. The prompt template includes variations in angle and lighting to ensure visual diversity.
Activation Extraction and Scoring: The generated images are processed by the target vision model to extract the specific neuron's activation vector. A scoring function ( $\psi_{avg}$ ) calculates the mean activation over the generated set. This continuous score quantifies the correctness of the proposed concept. The scoreboard is updated with the new concept and its score.

Final Summary Step: After $N$ iterations, a final summary step is executed. The LLM is prompted to synthesize a single, high-level global concept that unifies the top-scoring concepts from the final history, providing a broader semantic abstraction.

The method is architecture-agnostic, applicable to both Convolutional Neural Networks (CNNs) and Vision Transformers (ViTs), and supports various T2I models (SD1.5, SDXL, FLUX.1).

Key Contributions

The paper presents the following contributions:

State-of-the-Art Neuron Labeling: LINE achieves superior performance on the CoSy (Concept Synthesis) benchmark, a unified, architecture-agnostic benchmark for evaluating textual neuron explanations.
Beyond Predefined Vocabularies: The iterative refinement process allows LINE to discover novel, highly interpretable concepts that are missed by methods relying solely on predefined vocabularies.
Gradient-Free Visual Explanations: By optimizing textual descriptions rather than pixel values via gradients, LINE provides a gradient-free alternative to Activation Maximization. This enables the use of modern T2I models and produces cleaner, more natural visual explanations without the artifacts common in gradient-based AM.
Transparency and Polysemanticity Evaluation: Unlike single-pass methods, LINE maintains a complete generation history. This allows for the evaluation of neuron polysemanticity (neurons detecting multiple concepts) and provides reasoning behind each concept proposal.

Experimental Results

The authors evaluated LINE across multiple architectures (ResNet18/50, ViT-B/16, DINOv3) and datasets (ImageNet, Places365).

Quantitative Performance: On the CoSy benchmark, LINE significantly outperformed baselines including CLIP-Dissect, INVERT, and DnD.
- AUC Improvements: LINE achieved AUC improvements of up to 0.11 on ImageNet-trained models and 0.05 on Places365-trained models compared to the best baselines.
- MAD Improvements: Mean Activation Difference (MAD) scores were also consistently higher, indicating stronger alignment between the proposed concepts and neuron activations.
- Statistical Significance: Wilcoxon signed-rank tests confirmed that these improvements were statistically significant across most configurations.
Novel Concept Discovery: Analysis of the final top-scoring concepts revealed that LINE's iterative process generated new, non-predefined labels for a significant portion of neurons:
- 27% on average of the top concepts were newly discovered (not from the initial vocabulary).
- Specific layers showed discovery rates up to 36% (e.g., ResNet18 on Places365).
Qualitative and Causal Validation:
- Visual Explanations: LINE produced visual explanations that were more interpretable and natural than those from DiffExplainer and DEXTER. While DiffExplainer often achieved higher activation values, the authors attribute this to the exploitation of non-descriptive visual artifacts (noise) common in gradient-based optimization.
- Causal Ablation: Using image-to-image generative models, the authors performed ablation studies where the identified concept was removed from images. This resulted in a significant drop in neuron activation (often near zero), validating the causal link between the label and the neuron's behavior.
Ablation Studies:
- Iterations: Performance improved rapidly in the first 5–6 iterations, with steady gains up to iteration 14, after which it plateaued.
- T2I Models: While FLUX.1 produced slightly higher activation scores than SDXL, the difference was not statistically significant. SDXL was selected for the main experiments due to lower memory overhead.

Significance and Claims

The paper positions LINE as a foundational step toward more transparent and accountable vision models. Its significance lies in:

Training-Free and Black-Box Compatibility: It offers a scalable solution for auditing DNNs without requiring access to internal gradients, training data, or model weights, making it applicable to proprietary or legacy models.
Modularity and Future-Proofing: The framework's modular design allows for the seamless integration of future advancements in LLMs and T2I models without retraining the pipeline.
Safety and Interpretability: By providing both textual and visual explanations, LINE aids in identifying model biases and understanding decision-making circuits, which is critical for deploying AI in safety-critical applications.

The authors acknowledge limitations, including potential biases inherited from the ImageNet initialization and the reliance on generative models that may struggle with specialized domains (e.g., medical imaging). They also note that for polysemantic neurons, the pipeline may converge on a single dominant concept, suggesting future work could involve restarting the pipeline to explore secondary semantic triggers.