Technical Summary: TREETRACER – Exposing the Unsaid via Stochastic Path Aggregation

Problem Statement

Large Language Models (LLMs) generate text through stochastic processes, producing a probability distribution over possible next tokens rather than a single deterministic output. Current bias auditing methods face significant limitations in this context:

• Single-Output Insufficiency: Standard approaches often rely on inspecting a single generated sequence or static automated metrics (e.g., perplexity, BLEU). These methods obscure the underlying probability distributions and fail to capture biases hidden in lower-probability generation branches.
• Static Template Limitations: Foundational bias datasets often use rigid, predefined sentence templates. Research indicates these fail to capture the diversity of real-world contexts and syntactic variations where bias naturally manifests, often eliciting default syntactic behaviors rather than revealing true semantic associations.
• Scalability of Visualization: Existing visual analytics tools for text generation (e.g., Beam Search Trees) struggle to summarize hundreds of complex, stochastic generation paths into a condensed, readable view without losing structural details or cognitive clarity.

The core challenge is to develop a method that automates the generation and semantic classification of diverse test cases while enabling human evaluators to visually analyze aggregated probability distributions to detect systemic representational harms.

Methodology: TREETRACER Pipeline

The paper introduces TREETRACER, a visual analytics approach designed to evaluate LLM bias through aggregated comparison. The system employs a systematic perturbation pipeline consisting of six computational steps:

1. Prompt Augmentation and Ontology Injection:

   • The system isolates semantic variables in a base prompt by replacing a target token with substitutes drawn from a semantic ontology (e.g., "Male Names" vs. "Female Names").
   • Ontologies are generated via ConceptNet (BFS queries) or instruction-tuned LLMs.
   • A Human-in-the-Loop mechanism allows users to edit, remove, or create custom ontologies to prevent circular ambiguity and ensure human control over analytical parameters.

2. Text Generation and Token Alignment:

   • The system executes text generation for all augmented prompts using local inference or external APIs.
   • It records generated text, token-level probabilities, and metadata.
   • Subword Token Merging: To handle subword tokenizers (e.g., BPE), the pipeline merges subword tokens back into whole words, calculating the geometric mean of their probabilities to avoid penalizing longer words.

3. Constituency Parsing:

   • Generated outputs are parsed into constituency trees using the Stanza NLP library to create a unified grammatical representation.
   • An alignment algorithm maps the parser's part-of-speech tags back to the original whitespace tokens to resolve tokenization discrepancies.

4. Structural Clustering:

   • To address visual clutter, the system strips lexical content from trees to create "skeleton strings" (structural tags only).
   • These skeletons are clustered using agglomerative clustering with Tree Edit Distance (TED) or heuristics (Levenshtein, N-gram Jaccard).
   • The system retains the top-$k$ most frequent grammatical structures to form the visual backbone, while discarding low-frequency outliers to maintain visual clarity. A complete set of all sentences is retained separately for global probability calculations.

5. Semantic Classification:

   • Instead of static dictionaries, the system uses an LLM-Assisted Contextual Classification method. A helper LLM classifies filtered keywords (nouns, adjectives, proper nouns) within the full-sentence context to handle polysemy.
   • RAG-Based User Overrides: Users can manually correct misclassifications in the frontend. These corrections are saved and injected into the helper LLM's prompt for subsequent runs, ensuring deterministic, human-verified classification.

6. Tree Construction and Probability Metrics:

   • Nodes are merged using a composite key of (token_text, semantic_category) to prevent merging polysemous words.
   • The system computes two distinct probability metrics for visualization:
     • Selected Probability ($P_{selected}$): The average token confidence over the structurally retained sentences (visualized as link width).
     • Global Probability ($P_{global}$): The sum of token probabilities across all generated occurrences, normalized by the count of the selected subset. This ensures that tokens frequent in the total pool but pruned by clustering remain visually prominent (node height).

Key Contributions

1. Visual Analytics Interface (Sankey Adaptation)

TREETRACER utilizes a novel adaptation of the Sankey diagram to visualize aggregated probability trees. It decouples strict flow conservation rules to satisfy bias auditing needs:

• Node Height: Encodes $P_{global}$, representing the scaled probability mass of a token path across all samples.
• Link Width: Encodes $P_{selected}$, visualizing the model's localized confidence restricted to the selected structural subset.
• This dual encoding ensures that strong biases hidden within discarded syntactic variations remain visible.

2. Comparative and Contrastive Inference Modes

• Juxtaposition: Side-by-side visualization of two ontology-driven trees allows for direct comparison of structural divergence.
• Superposition: An interactive merged view aligns trees from different ontologies using a vertical cross-tree merge algorithm, preserving dual-origin statistics.
• Contrastive Inference: To address the "statistical blind spot" where decoding algorithms drop lower-probability tokens, the system performs counterfactual inference. It forces the model to calculate raw probabilities for specific tokens under both original conditions (Ontology A and B) and visualizes the Contrastive Split Ratio ($R_{split}$) as split beams within the edges.

3. System Validation

The system was validated through five case studies comparing an unaligned baseline (GPT-2 XL) against constitutionally aligned Apertus models:

• Gender Bias: Revealed hidden biases in occupational associations (e.g., female names linked to caregiving roles, male names to executive roles) and counterfactual pronoun suppression.
• Toxicity: Exposed historical training biases where Arab geographic appellations were linked to "Political & Religious Extremism" clusters, while Western appellations were linked to positive characteristics.
• Persona Roleplay: Demonstrated how roleplay constraints activate gendered stereotypes (e.g., female CEOs described via emotional labor vs. male CEOs via intelligence).
• Medical Disinformation: Showed how aligned models (Apertus) structurally collapse into hard refusal nodes for dangerous substances, whereas unaligned models (GPT-2) generate coherent but harmful instructions.
• Syntactic Bias: Highlighted "syntactic-domain spurious correlations" where unaligned models rigidly apply geographic templates to food items, whereas aligned models adapt semantically.

4. User Study

A preliminary user study with seven computer science students confirmed that the aggregated comparative interface:

• Reduces cognitive load compared to standard beam search tools (e.g., generAItor).
• Effectively supports analysts in detecting systemic biases.
• Achieved a System Usability Scale (SUS) score of 76.9, indicating high usability.

Significance and Claims

The paper claims that TREETRACER bridges the gap between stochastic text generation and the need for rigorous bias auditing. Its primary significance lies in shifting the analytical focus from isolated, single-instance inspection to aggregated, classification-aware comparison.

The authors assert that:

1. Hidden Biases are Exposed: Bias often resides in low-probability branches and syntactic variations that standard top-k or beam search inspections miss. TREETRACER makes these "hidden associations" explicit.
2. Ontology as a Control Variable: By treating ontologies as controlled independent variables, the system enables the separation of semantic effects from syntactic templates, offering a more robust tool for evaluating representational harms than static templates.
3. Counterfactual Rigor: The inclusion of contrastive inference reduces the risk of misinterpreting the presence of bias by computing and displaying counterfactual token probabilities, ensuring that the absence of a word in the final output does not mask its underlying probability.

The work concludes that combining visual aggregation, classification-aware merging, and contrastive inference provides a scalable basis for future AI safety and alignment research, allowing for the detection of systemic representational harms even when they are absent from top-ranked outputs.