💻 computer science

Interpreting Black-Box Large Language Models with Sentence-Level Energy Landscapes

This paper proposes a model-agnostic, post-hoc attribution interpreter that utilizes an Energy-Based Model as a surrogate to train a standalone tool capable of quantifying sentence-level prompt influence on LLM outputs without requiring further API queries.

Original authors: Maryam Rezaee, Pooriya Safaei, Maryam Asgarinezhad, Fatemeh Seyyedsalehi

Published 2026-08-05

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Maryam Rezaee, Pooriya Safaei, Maryam Asgarinezhad, Fatemeh Seyyedsalehi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are talking to a very smart, but mysterious, robot friend. You ask it a question, and it gives you a long, helpful answer. But here's the catch: you can't see inside the robot's brain. You can't peek at its gears or read its diary to see why it chose those specific words. This is the world of "Black-Box" Artificial Intelligence. These are powerful computer programs called Large Language Models (LLMs) that are so complex and secretive that even their creators sometimes can't explain exactly how they make decisions. This is a big problem if you want to trust the robot with important tasks, like giving medical advice or legal help. You need to know: "Did the robot say this because of that part of my question, or did it just make it up?"

To solve this, scientists have been trying to build "interpreters"—tools that act like X-ray glasses for AI. Usually, these tools try to look at the tiny building blocks of language, like individual words or letters (called "tokens"). But thinking about language one letter at a time is like trying to understand a movie by looking at single pixels; it's messy and often misses the bigger picture. The real magic happens in complete thoughts, or sentences. The big question this paper tackles is: Can we build a tool that ignores the tiny pixels and instead looks at whole sentences to figure out which parts of your question actually caused the robot's answer?

The researchers in this paper say "Yes, we can!" and they built a clever new way to do it. Instead of trying to open the black box, they built a "shadow twin" of the robot. Think of this twin as a detective who watches the real robot work, learns its habits, and then builds a map of its thinking process. They call this map an "Energy Landscape." Imagine a hilly terrain where low valleys represent "good, logical answers" and high peaks represent "weird, wrong answers." The real robot always tries to stay in the low valleys.

The team trained their detective twin to understand this landscape. Once the twin knows the map, it can look at a specific answer the robot gave and ask, "Which sentence in the question pushed the robot down into this specific valley?" They built a lightweight tool, which they call ESCI, that acts as a standalone translator. Once trained, this tool doesn't need to ask the big robot for help anymore; it can just look at the question and the answer and point directly to the most important sentences.

Here is how they tested it and what they found. They used a popular AI model (GPT-4o-Mini) as their "black box" and fed it thousands of questions and answers. They trained their ESCI tool to figure out which parts of the questions mattered most. When they compared ESCI's guesses to the guesses made by other super-smart AI models (acting as "oracles"), they found that ESCI was surprisingly accurate. It could spot the main point of a question even when there was a lot of extra chatter or "filler" words. For example, if you asked, "Explain like I'm five," ESCI correctly identified that the "Explain like I'm five" part was the most important instruction, not just the topic you were asking about.

However, the paper is careful not to claim this is a perfect, magic solution. The authors suggest that while ESCI is very good at finding the right sentences, it's not a crystal ball that sees the robot's exact internal thoughts. They measured this by doing a "what-if" test: they took the sentences ESCI said were important, removed the rest, and asked the robot to answer again. The robot was still able to give a very similar answer, which suggests ESCI found the true "causal" drivers. But, they also noted that if you remove the important sentences, the robot sometimes still guesses the right answer because it has so much general knowledge. This means ESCI is great, but it's not the final word on how the robot thinks.

One of the coolest things about this method is how efficient it is. Other methods that try to do this often have to ask the big robot thousands of questions just to figure out one answer, which is slow and expensive. The ESCI tool, once trained, can do its job instantly without asking the big robot for help at all. It's like training a guide dog once, and then the dog can lead you through the city forever without needing to call the trainer. The researchers found that after training on about 20,000 examples, their tool could handle new questions much faster than the old methods, saving a huge amount of time and computer power.

In the end, the paper suggests that looking at sentences instead of tiny words is a smarter way to understand AI. It shows that we can build tools that help us trust these mysterious robots by pointing out exactly which parts of our conversation matter. While it's not a perfect window into the robot's soul, it's a very strong pair of glasses that helps us see the logic behind the magic, making it safer and more reliable to use these powerful tools in our daily lives.

Technical Summary: Interpreting Black-Box Large Language Models with Sentence-Level Energy Landscapes

Problem Statement

The widespread adoption of proprietary Large Language Models (LLMs) accessed exclusively through closed APIs has created a critical barrier to responsible deployment: a fundamental lack of interpretability. While white-box interpretability methods (e.g., gradient-based attribution, attention visualization) exist, they are incompatible with closed systems. Furthermore, standard model-agnostic post-hoc attribution techniques struggle with generative models due to the high-dimensional, interactive nature of text outputs and the ill-posed nature of explaining generation. Existing alternatives, such as prompt-based self-explanation (e.g., Chain-of-Thought), often suffer from circular logic and produce unfaithful confabulations rather than revealing true internal computation paths. The core challenge is to identify which specific parts of a user prompt drive a specific subset of an LLM's output without access to the model's internal weights or gradients.

Methodology

The authors propose ESCI (Energy-Based Concept-Level Surrogate Interpreter), a model-agnostic, post-hoc attribution framework that operates at the sentence level rather than the token level. The methodology consists of two primary stages:

1. Sentence-Level Abstraction

The framework treats sentences as "concepts"—the smallest linguistic units expressing a complete proposition. Input prompts ( $x$ ) and LLM responses ( $y$ ) are segmented into sentences and embedded using a frozen, pre-trained Sentence-BERT model. This shifts the analysis from noisy tokens to coherent semantic units, aligning with the "Context Principle" where individual tokens are semantically ambiguous without propositional structure.

2. The Energy-Based Surrogate (EBM)

To approximate the black-box LLM's behavior, the authors train a transformer-based Energy-Based Model (EBM), denoted as $E_{LM}(x, y; \theta)$ .

Architecture: The EBM processes sentence embeddings through a "concept space" projection (using self-attention) and an input-output interaction module (using cross-attention). It outputs a scalar energy value representing the compatibility between a prompt and a response.
Training Objectives: The EBM is trained using a dual-objective contrastive loss:
- Fidelity ( $L_{fidelity}$ ): An InfoNCE loss that distinguishes authentic prompt-response pairs from global negatives (e.g., human answers or outputs from other LMs) to capture the target LLM's global generation signature.
- Local Dependency ( $L_{dep}$ ): A loss that enforces local causal precision by contrasting partial prompts against mismatched partial responses (and vice versa), ensuring the model learns specific input-output dependencies rather than surface artifacts.
Surrogate Role: The EBM acts as a differentiable surrogate, learning the shape of the target LLM's generation manifold. Low energy indicates high consistency with the target model's conceptual interactions.

3. The Interpreter Network

A lightweight interpreter network, $I_N(x, y_T; \alpha)$ , is trained to identify the subset of prompt sentences ( $x_S$ ) most influential in generating a user-specified target output subset ( $y_T$ ).

Mechanism: The interpreter takes the prompt, the full response, and the target subset as input. It outputs a sparse, binary importance vector.
Training via Alternating Optimization: The interpreter is trained to maximize the energy gap between the selected prompt subset (which should yield low energy when paired with the target) and the unselected remainder (which should yield high energy). To prevent distribution shifts caused by masking inputs, the framework employs alternating optimization: the interpreter is updated, and periodically, the EBM is fine-tuned using fresh prompt-response pairs generated by the target LLM based on the interpreter's current selections.
Inference: Once trained, the interpreter operates as a standalone tool. It requires zero additional API queries to the target LLM during inference, making it highly scalable.

Key Contributions

Concept-Level Attribution: The work shifts the unit of analysis from tokens to sentences, enabling human-intelligible attribution that relates output elements directly to user prompt concepts.
Transformer-Based EBM Surrogate: The authors introduce a novel EBM architecture with specific sampling methods to learn a surrogate random field over prompts and responses. This approach models authentic input-output dynamics without requiring normalized probability distributions over vast output spaces.
Model-Agnostic Framework: The proposed framework provides a post-hoc interpreter for black-box LLMs that identifies specific prompt sentences responsible for triggering target outputs, operating independently of the target model's internal architecture.

Experimental Results

The framework was evaluated on the HC3 multi-domain Q&A dataset, using GPT-4o-Mini as the target black-box LLM.

Ablation Study: The dual-objective EBM ( $M_{Hybrid}$ ) outperformed single-objective variants. It achieved superior Directive Localization (isolating the primary query from noise), Semantic Robustness (ignoring conversational fillers), Oracle Alignment (ranking consistency with state-of-the-art models like Gemini-2.5-Flash), and Causal Disentanglement (distinguishing true antecedents from distractors).
Attribution Plausibility: When compared against five LLM "oracles" (including GPT-4o and GPT-J-6B), ESCI demonstrated competitive alignment. Notably, ESCI produced sparse, confident attribution scores, effectively isolating necessary dependencies, whereas instruction-tuned oracles often produced diffuse distributions. In qualitative case studies, ESCI successfully disentangled stylistic drivers (e.g., "Explain like I'm five") from semantic content, a capability where other oracles sometimes failed.
Causal Faithfulness: Using metrics of Generative Sufficiency (can the target be regenerated with only selected sentences?) and Generative Comprehensiveness (is the target lost when selected sentences are removed?), ESCI achieved a positive Faithfulness Gap comparable to strong baselines like GPT-4o and sentence-level LIME.
Scalability: A critical finding is the computational efficiency. While LIME requires dozens of API queries per prompt for perturbation-based analysis, ESCI operates in $O(1)$ inference time with zero additional API calls. The framework breaks even in API costs after interpreting approximately 400 prompts, offering a scalable pathway for high-volume deployment.

Significance and Claims

The paper claims that ESCI establishes energy-based surrogates as a scalable pathway for diagnosing model behaviors in black-box settings. By modeling generation dynamics as a differentiable energy landscape, the framework enables the training of a standalone interpreter that requires no further interaction with the target LLM after pre-training.

The authors position this work as a solution to the opacity of proprietary LLMs in high-stakes domains. They argue that by capturing global generation patterns and mitigating instance-specific biases through global training, the framework provides a robust, concept-level explanation that is both computationally efficient and causally faithful.

Limitations Acknowledged:
The paper modestly acknowledges several limitations:

Training Overhead: The framework shifts computational burden to the pre-training phase, requiring task-specific training and hyperparameter tuning, which limits out-of-the-box generalizability.
Lack of Ground Truth: As a black-box setting, validation relies on probabilistic oracles rather than deterministic ground truth (e.g., internal gradients from open-weights models).
Scope: Current validation is limited to Q&A tasks with compact surrogate models; performance on complex reasoning or massive context windows remains to be verified.
Human Utility: While causal faithfulness is quantified, the practical utility for human auditing tasks (e.g., bias detection) requires further subject studies.