🤖 AI

LLMbench: A Comparative Close Reading Workbench for Large Language Models

This paper introduces LLMbench, a browser-based workbench designed for the digital humanities that facilitates the comparative close reading of large language model outputs by visualizing token-level log-probabilities and probabilistic structures through unique analytical overlays and modes, thereby treating generated text as a critical research object to reveal its counterfactual possibilities.

Original authors: David M. Berry

Published 2026-04-20

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: David M. Berry

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are reading a story written by two different AI robots. You ask them both, "Tell me about this old book," and they both write a paragraph. To a normal reader, you just see the final words on the page. But to a computer scientist, those words are just the tip of the iceberg.

This paper introduces LLMbench, a special digital "workbench" (like a high-tech microscope) that lets researchers look under the hood of how AI writes. Instead of just reading the story, it lets you see the ghosts of the stories that almost happened.

Here is a simple breakdown of how it works, using some everyday analogies:

1. The Core Idea: The "Road Not Taken"

When an AI writes a sentence, it doesn't just "know" the next word. It's actually rolling a weighted dice.

The Analogy: Imagine you are walking down a path. At every step, the AI looks at a fork in the road. It might see 10 different paths. 9 of them are muddy and unlikely, but 1 is very clear. It picks the clear one.
The Problem: Most tools only show you the path the AI chose. They don't show you the 9 muddy paths it ignored.
The Solution: LLMbench shows you all the paths. It reveals the "probability distribution"—the map of every word the AI considered and how likely it was to pick each one.

2. The Main Feature: The "Heat Map" of Uncertainty

The tool puts two AI responses side-by-side (like comparing two students' essays). But it adds a special "Heat Map" overlay.

The Analogy: Think of the text as a road.
- Blue/Green areas: The AI was 100% sure. It was walking on a paved highway. "The cat sat on the..." -> It knew the next word was "mat."
- Red/Orange areas: The AI was confused or guessing. It was standing at a crossroads with 50 different signs. "The cat sat on the..." -> It was torn between "mat," "floor," "chair," "rug," and "sofa."
Why it matters: If you see a red spot, it means the AI was "rolling the dice" there. If you see a blue spot, it was just reciting a fact. This helps researchers find where the AI is being creative (or hallucinating) versus where it is just repeating patterns.

3. The "Time Machine" Views

The tool has different ways to visualize this data, like looking at a landscape from different angles:

The Sparkline (Graph): A squiggly line that shows how "confused" the AI was as it wrote the story. High peaks mean the AI was struggling to decide; flat lines mean it was cruising.
The Pixel Map: A grid of colored dots. It's like looking at a city from a helicopter. You can instantly see which parts of the essay were "foggy" (uncertain) and which were "clear."
The 3D Terrain: Imagine the text is a mountain range. The flat plains are where the AI was confident. The jagged, high peaks are where the AI was unsure. You can rotate this 3D map to see the "shape" of the AI's thinking.

4. Comparing Two Robots

The tool lets you compare two different AIs (like Gemini vs. GPT-4) answering the same question.

The Analogy: Imagine two chefs making the same soup.
- Chef A is very confident about the salt but guesses wildly on the herbs.
- Chef B is unsure about the salt but very confident about the herbs.
- LLMbench shows you exactly where their confidence differs. Maybe one chef is "rolling the dice" on a crucial word, while the other is just following a recipe.

5. Why Do We Need This?

Currently, we mostly judge AI by scores (like a test grade) or by asking humans "Which answer is better?"

The Paper's Argument: This is like judging a painter only by the final picture. LLMbench lets us study the brushstrokes.
It helps us understand:
- Where the AI is lying or guessing.
- Where the AI is being creative.
- How the AI's "personality" (its training) changes the way it makes choices.

Summary

LLMbench is a tool that turns AI writing from a "black box" (where we only see the result) into a glass box (where we can see the gears turning). It treats the AI's output not just as a finished text, but as a snapshot of a million "what-ifs" that happened in a split second before the word was chosen.

It allows researchers to say: "Ah, the AI didn't just choose this word because it's the best one; it chose it because it was the only one left on the table after rolling the dice." This helps us understand AI not just as a tool, but as a complex, probabilistic system that we can actually read and critique.

Based on the paper "LLMbench: A Comparative Close Reading Workbench for Large Language Models" by David M. Berry, here is a detailed technical summary:

1. Problem Statement

The proliferation of Large Language Models (LLMs) has led to an abundance of evaluation tools, but these are predominantly quantitative and engineering-focused. Existing tools (e.g., Google PAIR's LLM Comparator, LMSYS Chatbot Arena, HELM) rely on user ratings, win-rates, and leaderboard metrics to evaluate model performance.

The Gap: These tools treat the LLM output as a fixed object for evaluation rather than a probabilistic object generated from a distribution. They lack the capacity for "close reading" (a hermeneutic practice from the Digital Humanities) that investigates the probabilistic structure of the text—specifically, the "roads not taken" (alternative tokens) and the uncertainty inherent in the generation process.
The Need: There is a need for a workbench that treats generated text as a research object in its own right, allowing researchers to inspect the log-probability data (logprobs) to understand the counterfactual history of the text (i.e., what the text could have been).

2. Methodology and System Architecture

LLMbench is a browser-based workbench designed for the comparative close reading of two LLM outputs side-by-side. It bridges the gap between engineering evaluation and Digital Humanities (DH) textual analysis.

Technical Stack: Built with Next.js 16 and React 19, deployed on Vercel. It uses the Vercel AI SDK for a unified interface across providers (Anthropic, OpenAI, Google, Hugging Face, OpenRouter).
Interface: Utilizes CodeMirror 6 (a code-editor substrate) rather than a word processor. This provides a "defamiliarized" view of text, treating it as an object for analysis, and supports gutter-based annotation.
Data Source: The core methodology relies on log-probability data (logprobs) returned by model APIs. This data reveals the top- $k$ candidate tokens and their probabilities at every generation step.
Annotation Framework: Adapts the Critical Code Studies (CCS) methodology (Observation, Question, Metaphor, Pattern, Context, Critique) for annotating LLM outputs.

3. Key Features and Modes

The tool operates through two primary sections: Compare Mode (qualitative close reading) and Analyse Modes (quantitative empirical probes).

A. Compare Mode (Side-by-Side Analysis)

Two model responses to the same prompt are displayed in synchronized panels with four analytical overlays:

Probabilities (Probs): The core analytical layer. It overlays a continuous heatmap on the text based on token confidence.
- Visuals: Uncolored (high confidence) to deep red (high entropy/uncertainty).
- Navigation: Buttons to jump to "Uncertain" (high entropy), "Forks" (low probability choices), and "Diverge" (where models chose different tokens).
- Inspector: Clicking a token reveals the full probability distribution of alternatives, allowing users to see the "counterfactuals" (e.g., if the model chose "concerning" at 11% probability, what were the other 89% options?).
- Visualizations: Includes Graph (entropy sparkline), Pixels (2D heat map of the whole text), and Net (3D terrain of entropy).
Differences (Diff): Highlights word-level differences between the two panels, showing unique vocabulary and phrasing variations.
Structure (Struct): Breaks text into numbered sentences and highlights discourse connectives (e.g., "however," "therefore") to reveal the argumentative arc and syntactic structure.
Tone: Applies Ken Hyland's metadiscourse model to categorize rhetorical stance (Hedges, Boosters, Attitude markers, Self-mentions, etc.), visualizing the model's confidence and engagement style.

B. Analyse Modes (Empirical Probes)

Five modes run automated experiments to quantify model behavior:

Stochastic Variation: Runs the same prompt multiple times (3–20) to measure output variance (word overlap, vocabulary diversity).
Temperature Gradient: Runs the prompt across temperatures (0.0 to 2.0) to observe how randomness affects output length, diversity, and coherence.
Prompt Sensitivity: Auto-generates minor prompt variations (e.g., adding "please," changing punctuation) to test how fragile the model's response is to slight input changes.
Token Probabilities: A standalone mode for deep inspection of a single response's entropy distribution and token-level confidence.
Cross-Model Divergence: Quantifies the difference between two models using Jaccard similarity, word overlap, and structural metrics (sentence length, vocabulary diversity).

4. Results and Case Study

The paper demonstrates the tool using a comparison of Gemini 2.0 Flash and GPT-4o responding to a prompt about Italo Calvino's Cybernetics and Ghosts.

Findings:
- Divergent Uncertainty: At the same token position (e.g., position 26), Gemini showed high entropy (2.315 bits, 11.76% confidence) with multiple near-equivalent alternatives, while GPT-4o showed lower entropy (1.567 bits, 49.27% confidence) with a tighter distribution.
- Structural Differences: The "Diverge" navigation revealed 281 points where the models chose different tokens, often corresponding to different rhetorical strategies (e.g., one model continuing a sentence vs. the other ending it).
- Lexical Divergence: Cross-Model Divergence metrics showed a Jaccard similarity of only 22.8%, indicating the models drew from largely different lexical territories despite the same prompt.
- Hermeneutic Insight: The tool revealed that "position 26" meant different discursive things for each model because they structured their opening paragraphs differently, a nuance invisible to standard text comparison.

5. Limitations

API Constraints: Log-probability data is not uniformly available. Anthropic's Claude models (at the time of writing) and some open-weight models via OpenRouter do not expose this data, limiting the "Probs" overlay to a subset of models.
Tokenization Artifacts: Different models use different tokenizers. A "position 26" in one model does not necessarily align semantically with "position 26" in another, complicating direct token-level comparison.
Model Volatility: Commercial models are "moving targets" (weights and filters update without notice), making comparisons historical snapshots rather than permanent truths.
Methodological Fit: The Hyland metadiscourse framework was designed for academic writing; its application to AI-generated text is experimental and may not capture specific AI rhetorical tropes (e.g., "As an AI language model...").

6. Significance and Contribution

New Hermeneutic Practice: LLMbench introduces Comparative AI Hermeneutics, a reading practice that simultaneously analyzes the realized text and the probability distribution from which it emerged.
Critical Resource: It argues that log-probability data is an underutilized resource for critical AI studies, offering a way to see "what could have been" and distinguishing between deterministic and contingent elements in AI generation.
Bridging Traditions: It successfully merges the Digital Humanities tradition of close reading/annotation with the Engineering tradition of probabilistic model inspection.
Theoretical Impact: By visualizing the "counterfactual history" of text, the tool challenges the notion of LLM output as a fixed artifact, framing it instead as a selection from a cloud of near-equivalent alternatives, thereby deepening the critical understanding of generative AI.