💻 computer science

UNBOX: Unveiling Black-box visual models with Natural-language

The paper introduces UNBOX, a novel framework that leverages large language models and text-to-image diffusion models to dissect black-box visual classifiers under fully data-free and gradient-free constraints, generating human-interpretable text descriptors that reveal a model's learned concepts and potential biases without requiring access to its internal architecture or training data.

Original authors: Simone Carnemolla, Chiara Russo, Simone Palazzo, Quentin Bouniot, Daniela Giordano, Zeynep Akata, Matteo Pennisi, Concetto Spampinato

Published 2026-03-10

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Simone Carnemolla, Chiara Russo, Simone Palazzo, Quentin Bouniot, Daniela Giordano, Zeynep Akata, Matteo Pennisi, Concetto Spampinato

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a mysterious, high-tech vending machine (the AI model). You put in a picture of a dog, and it spits out a receipt saying "99% chance this is a dog." But here's the catch: the machine is a black box. You can't see inside. You don't know how it was built, what it was taught, or why it thinks a picture of a dog is a dog. It could be looking at the fur, the ears, or maybe it's just guessing because the dog is standing on grass.

For a long time, if you wanted to understand why this machine made that decision, you needed to take the machine apart (access its internal code/weights) or have a list of every single item it ever learned from (training data). If you didn't have those, you were stuck in the dark.

Enter UNBOX.

The paper introduces UNBOX, a clever new way to figure out what's inside that black box without ever opening it. It's like being a detective who can only ask the vending machine, "Is this a dog?" and listen to the confidence score on the receipt.

How Does UNBOX Work? (The Detective's Toolkit)

Instead of trying to reverse-engineer the machine's math, UNBOX uses a team of AI detectives (Large Language Models) and a magical sketch artist (a text-to-image generator). Here is the step-by-step process, explained with a simple analogy:

1. The "Guess and Check" Game

Imagine you are trying to describe a "Dog" to the sketch artist so they draw a picture that makes the vending machine scream, "YES! That is definitely a dog!"

The Detective (LLM): Starts with a vague guess: "A picture of a furry animal."
The Sketch Artist (Diffusion Model): Draws a generic dog.
The Vending Machine (Black Box): Looks at the drawing and says, "Hmm, 60% confidence."
The Detective: "Okay, we need to do better. Let's try adding 'brown' and 'wagging tail'."

2. The "Trend and Intensity" Compass

The machine doesn't give the detective a "why" or a "how." It only gives a number (the confidence score). UNBOX has a special trick to turn that number into a direction.

Trend: Is the score going up or down? If it went from 60% to 70%, the detective knows, "Great! Keep going in this direction!" If it dropped, they know, "Backtrack!"
Intensity: How far are we from a perfect 100%? If we are at 99%, the detective makes tiny, precise adjustments (micro-changes). If we are at 10%, they make big, bold changes (coarse adjustments).

3. The "Memory Bank" (Global & Local Context)

Sometimes, the detective gets confused or forgets what worked. UNBOX keeps two notebooks:

The Global Notebook: "Hey, remember that time we drew a 'snowy sled' and the machine loved it? Let's keep 'snow' and 'sled' in our memory, even if we aren't using them right now." This prevents the detective from forgetting the most important clues.
The Local Notebook: "We just tried 'blue fur' and it failed. Let's not try 'blue fur' again immediately." This stops the detective from spinning in circles.

4. The Final Reveal

After hundreds of rounds of guessing, drawing, and refining, the detective ends up with a perfect description.

Result: "A fluffy, brown dog with a wagging tail running on green grass."
The Twist: If the machine was actually biased (e.g., it only recognizes dogs on grass), the description might end up being: "A dog on a grassy field." This reveals a bias the machine has!

Why Is This a Big Deal?

1. It Works on "Closed" Machines:
Most previous methods were like trying to fix a car engine while wearing blindfolded gloves, but you were allowed to take the engine apart. UNBOX works even if the car is locked in a garage and you can only listen to the engine sound. This is crucial because many companies (like big tech firms) sell AI as a service where you can't see the code.

2. It Finds the "Secret Shortcuts":
AI models often cheat. They might think "a picture of a beach" means "a dog" because most dog photos in their training data were taken at the beach.

UNBOX exposes this: If you ask UNBOX to describe a "Dog," and it keeps coming up with "Dog on a beach," you now know the AI is biased. You can fix it before it causes problems in the real world.

3. It's Surprisingly Good:
The paper tested UNBOX on famous datasets (like ImageNet). Even though it had no internal access, it figured out what the AI was thinking almost as well as methods that did have access to the internal code.

The Bottom Line

UNBOX is a "Black Box Translator."

It takes the silent, hidden logic of a mysterious AI and translates it into human language. It doesn't just tell you what the AI sees; it tells you how the AI sees it, including its mistakes and biases.

Think of it as a mirror that you can hold up to a black box. Even though you can't see inside the box, the reflection shows you exactly what the box is thinking, allowing us to build AI that is fairer, safer, and more trustworthy.

1. Problem Statement

The paper addresses a critical gap in AI trustworthiness and interpretability within the context of modern proprietary visual recognition systems.

The Black-Box Constraint: Increasingly, visual models are deployed as closed APIs where users only have access to output probabilities. Internal details such as architecture, weights, gradients, training data, and activation maps are hidden.
Limitations of Existing Methods: Current explainability techniques (e.g., saliency maps, concept activation vectors, neuron dissection) generally require white-box (full weight/gradient access) or gray-box (partial access) capabilities. They often rely on probing datasets or internal activations, rendering them unusable for strict black-box auditing.
The Challenge: How can one perform global, class-wise model dissection—uncovering what concepts a model has learned, its training distribution, and potential biases—using only the scalar output probabilities of a target class?

2. Methodology: The UNBOX Framework

UNBOX (Unveiling Black-box visual models with Natural-language) reframes the problem of activation maximization from a gradient-based optimization in feature space to a semantic search in text space. It operates under three strict constraints: data-free, gradient-free, and backpropagation-free.

Core Components

The framework utilizes a closed-loop optimization process involving a Black-Box Classifier ( $\phi$ ), a Text-to-Image Diffusion Model ( $G$ ), and two cooperating Large Language Model (LLM) agents.

Semantic Optimization Loop:
- Initialization: Start with a natural language prompt $p_t$ .
- Synthesis: Generate a synthetic image $x_t = G(p_t)$ using a diffusion model.
- Evaluation: Query the black-box classifier to get the target class probability $s_t = \phi(x_t)_j$ .
- Signal Generation: Since no gradients exist, UNBOX computes two metrics from the temporal evolution of $s_t$ $s_{t}$ :
  - Trend ( $T_t$ ): Determines if the prompt is improving (up), deteriorating (down), or stagnating (flat) compared to an exponential moving average of past scores.
  - Intensity ( $I_t$ ): Measures the distance from a high-confidence threshold, dictating the magnitude of the required update (strong, moderate, or micro).
- Agent Refinement:
  - Feedback Agent ( $A_f$ ): Receives the current prompt, the semantic signal (Trend/Intensity), and optimization contexts. It generates a structured natural language critique.
  - Updater Agent ( $A_u$ ): Applies the critique to refine the prompt into $p_{t+1}$ , ensuring linguistic fluency and semantic coherence.
Optimization Contexts (Stabilization):
To prevent the optimization from collapsing into narrow or transient descriptors, UNBOX maintains two memory structures:
- Global Optimization Context: Accumulates high-scoring prompts ( $P_{best}$ ) and extracts atomic lexical units ( $D_{best}$ ) that consistently yield high activation. This preserves the full semantic spectrum of the class.
- Local Optimization Context: Stores recent prompt-feedback pairs to prevent oscillation and ensure smooth progression through the search space.
Termination: The loop continues until convergence or a step limit is reached. The final output is a ranked set of human-interpretable text descriptors that maximally activate the target class.

3. Key Contributions

First Fully Black-Box Dissection Framework: UNBOX is the first method to perform global, class-wise model dissection using only output probabilities, without access to weights, gradients, or training data.
Text-Space Activation Maximization: It successfully reformulates activation maximization as a semantic optimization problem driven by LLM-mediated feedback and diffusion-based evidence, bypassing the need for differentiable model internals.
Robust Optimization Mechanism: The introduction of Trend/Intensity signals and Global/Local Contexts allows the system to navigate the non-convex text space effectively, avoiding local optima and semantic drift.
Bias Discovery Capability: The method can uncover spurious correlations (e.g., background biases) by analyzing the descriptors and generated images, enabling downstream debiasing without training data.

4. Experimental Results

UNBOX was evaluated on ImageNet-1K, Waterbirds, and CelebA against state-of-the-art white-box (CLIPDissect, WWW) and gray-box (DEXTER) methods.

Semantic Fidelity: UNBOX achieved high similarity between recovered descriptors and ground-truth class labels.
- On ResNet50, it scored 0.64, outperforming the white-box WWW method (0.60) and competing with CLIPDissect (0.77).
- On ViT architectures, UNBOX achieved 0.49, significantly outperforming CLIPDissect (0.27) and WWW (0.40), demonstrating superior generalization across different model architectures.
Latent Training Semantics Recovery: The descriptors recovered by UNBOX aligned strongly with data-derived keywords (extracted from training image captions) and visual evidence in validation images, proving it captures the model's actual learned distribution rather than just plausible text.
Slice Discovery & Debiasing:
- UNBOX successfully identified spurious correlations (e.g., "water" for waterbirds, "branch" for landbirds).
- Using these descriptors to retrain classifiers via Distributionally Robust Optimization (DRO), UNBOX achieved 88.6% worst-group accuracy on Waterbirds and 90.01% on CelebA.
- These results are competitive with or approach the performance of methods that require full access to model internals and training data (e.g., Oracle DRO).
Ablation Studies: Removing the Global Context or Intensity state led to significant performance drops, confirming their necessity for stable convergence.

5. Significance and Impact

Democratizing Auditing: UNBOX enables meaningful auditing, bias detection, and failure analysis for proprietary models (e.g., commercial APIs) that were previously considered "uninterpretable."
Architecture Agnostic: The method works equally well on CNNs (ResNet) and Transformers (ViT), proving that output probabilities alone contain sufficient information to reconstruct class semantics.
Trustworthy AI: By revealing the implicit reasoning, training distribution biases, and spurious cues of black-box models, UNBOX provides a pathway toward more transparent and accountable visual recognition systems in open-world settings.
New Paradigm: It establishes text-space semantic optimization as a viable alternative to gradient-based methods for model interrogation, opening new research directions in black-box interpretability.

In conclusion, UNBOX demonstrates that the "black box" is not impenetrable; by leveraging the reasoning capabilities of LLMs and the generative power of diffusion models, one can reconstruct the internal logic of a visual classifier using only its final output.