💬 NLP

LADDER: Language-Driven Slice Discovery and Error Rectification in Vision Classifiers

LADDER is a novel framework that leverages large language models to discover coherent, language-driven error slices and generate pseudo-attributes for comprehensive bias mitigation in vision classifiers, overcoming the limitations of traditional clustering and attribute-based methods by integrating domain knowledge and advanced reasoning without requiring explicit annotations.

Original authors: Shantanu Ghosh, Rayan Syed, Chenyu Wang, Vaibhav Choudhary, Binxu Li, Clare B. Poynton, Shyam Visweswaran, Kayhan Batmanghelich

Published 2026-08-07

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Shantanu Ghosh, Rayan Syed, Chenyu Wang, Vaibhav Choudhary, Binxu Li, Clare B. Poynton, Shyam Visweswaran, Kayhan Batmanghelich

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are teaching a robot to recognize animals. You show it thousands of pictures of cats and dogs, and it gets really good at guessing. But then, you notice something weird: the robot isn't actually learning what a cat is. Instead, it is relying on shortcuts. It's looking at the background. If the cat is on a green lawn, the robot thinks, "Green lawn? Must be a cat!" But if you show it a cat on a red carpet, the robot panics and guesses "dog." This is called a "bias," and it's a sneaky problem in artificial intelligence. The robot is relying on shortcuts instead of doing the hard work of understanding.

For a long time, scientists tried to fix this by making a giant checklist of things to look for, like "is there grass?" or "is there a tree?" But this is like trying to catch a thief by only checking for red hats; what if the thief is wearing a blue one? The old methods were too rigid. They couldn't think outside the box, and they often missed the subtle clues that made the robot fail. This is where a new idea comes in: what if we could just ask the robot, "Hey, why did you get that wrong?" and have it explain itself using words? That's the big question this paper tackles.

Enter LADDER, a clever new tool that acts like a detective for AI mistakes. Instead of forcing the robot to fit into a pre-made checklist, LADDER uses a super-smart language brain (called a Large Language Model, or LLM) to figure out why the robot is failing. Think of it this way: if the robot is a student taking a test, the old methods were like a teacher who only checked if the student wrote the right answer. LADDER is like a teacher who reads the student's scratch paper, sees they were distracted by a bird outside the window, and says, "Ah! You didn't fail because you don't know math; you failed because you were looking at the bird!"

Here is how LADDER works its magic. First, it looks at the pictures the robot got right and the ones it got wrong. It then asks a language expert (the LLM) to read the descriptions or "captions" of those pictures. The language expert is like a detective with a magnifying glass, scanning the text for clues. It might say, "Wait a minute! Every time the robot got the 'pneumothorax' (a lung condition) diagnosis right, the report mentioned a 'chest tube.' But when it got it wrong, the chest tube was missing!" The robot wasn't looking at the lung; it was just looking for the tube.

Once LADDER spots these sneaky patterns, it doesn't just point them out; it fixes them. It creates a new set of rules (called "pseudo-labels") to teach the robot to ignore the chest tubes and actually look at the lungs. It's like telling the student, "Next time, ignore the bird and focus on the math problem." The paper tested this on all sorts of tricky tasks, from spotting birds in forests to finding cancer in medical scans. They tried it on over 200 different robot brains and found that LADDER was much better at finding these hidden mistakes than the old checklist methods.

The best part? LADDER doesn't need a human to write down every single thing it should look for. It figures it out on its own by reading the text that comes with the images. In fact, when they tested it on medical images, LADDER found biases that even the old methods missed, like the robot getting confused by the age of the patient or the specific type of machine used to take the X-ray. The paper suggests that by using this language-driven detective work, we can make AI much fairer and more reliable, especially in important fields like medicine where getting the answer right matters the most. It's not a magic wand that solves everything instantly, but it's a powerful new way to listen to our AI and help it learn the right lessons.

Technical Summary: LADDER – Language-Driven Slice Discovery and Error Rectification in Vision Classifiers

1. Problem Statement

Vision classifiers often exhibit systematic failures on specific data subsets, known as error slices, due to spurious correlations (biases) learned during training. Identifying these slices is critical for improving model robustness, particularly in specialized domains like radiology where biases can be subtle and domain-specific.

Current slice discovery methods face three primary limitations:

Restricted Attribute Banks: They rely on predefined sets of attributes or visual tags, limiting their ability to discover novel or complex biases.
Lack of Reasoning and Domain Knowledge: They often lack the common sense or specialized domain knowledge (e.g., radiological pathology) required to interpret complex error patterns.
Inability to Capture Preprocessing Biases: They typically overlook biases introduced during data preparation or preprocessing (e.g., metadata, DICOM headers) that are not present in the raw image pixels.

Furthermore, existing mitigation strategies often require expensive ground-truth annotations for bias attributes or assume prior knowledge of the number and types of biases, which is rarely available in real-world scenarios.

2. Methodology: LADDER

LADDER (Language-Driven Slice Discovery and Error Rectification) is a framework that leverages the reasoning capabilities and latent domain knowledge of Large Language Models (LLMs) to automatically discover and mitigate error slices without requiring explicit attribute annotations. The method operates in three main stages:

A. Retrieving Sentences Indicative of Biases

LADDER begins by analyzing the internal activations of a pre-trained classifier $f$ .

Projection: It learns a projection function $\pi$ to align the classifier's image representation space ( $\Phi$ ) with a Vision-Language Representation (VLR) space ( $\Psi_I$ ), utilizing models like CLIP or domain-specific variants (e.g., CXR-CLIP).
Difference Estimation: For a specific class, it calculates the difference ( $\Delta_I$ ) between the mean projected representations of correctly classified samples and misclassified samples.
Retrieval: This difference vector is used to retrieve the top- $K$ sentences from a text corpus (e.g., image captions, radiology reports, or metadata logs) that are most semantically similar to the error pattern. These sentences describe visual attributes present in correct predictions but missing in errors.

B. Discovering Error Slices via LLM

Instead of relying on unsupervised clustering, LADDER invokes an LLM to analyze the retrieved sentences.

Hypothesis Generation: The LLM generates a set of testable hypotheses ( $H$ ) regarding potential biases (e.g., "The classifier relies on the presence of chest tubes").
Prompt Construction: For each hypothesis, the LLM generates specific text prompts ( $T$ ) that describe the attribute in diverse contexts.
Slice Identification: The system computes the similarity score ( $s_H$ ) between an image's projected representation and the mean embedding of the hypothesis-specific text prompts. Images with low similarity scores (indicating the absence of the hypothesized attribute) are flagged as a potential error slice. If the error rate on this subset is significantly higher than the overall class error, it is confirmed as an error slice.

C. Mitigation without Annotation

LADDER mitigates biases without manual labeling:

Pseudo-Labeling: It converts the similarity scores into probabilities to assign pseudo-labels (presence/absence of the bias) to the training data.
Ensemble Fine-tuning: Following the Deep Feature Re-weighting (DFR) approach, it creates balanced datasets for each hypothesis and fine-tunes the classifier's linear head.
Inference: During inference, the system selects the debiased classifier head associated with the hypothesis that yields the maximum similarity score for the input image, effectively ensembling multiple debiased models.

3. Key Contributions

The paper outlines five primary contributions:

Language-Driven Discovery: Utilizing natural language (captions, reports) to retrieve sentences indicative of errors, capturing insights beyond simple attribute presence/absence.
LLM Reasoning and Domain Knowledge: Leveraging LLMs to generate hypotheses and apply latent domain knowledge (e.g., identifying specific disease subtypes or pathological patterns) to detect nuanced biases that tagging models miss.
Model Agnosticism: The ability to detect slices from any off-the-shelf supervised model, regardless of architecture or pretraining strategy, overcoming the specific training requirements of methods like Facts or DrML.
Beyond Captions: Extending bias detection to metadata (e.g., Electronic Health Records, DICOM headers) by converting structured data into text for LLM analysis.
Annotation-Free Mitigation: Mitigating multiple biases simultaneously by generating pseudo-labels and fine-tuning, requiring no prior knowledge of the number or type of biases.

4. Experimental Results

The authors evaluated LADDER on 6 datasets (3 natural image: Waterbirds, CelebA, MetaShift; 3 medical imaging: NIH Chest-X-ray, RSNA-Mammo, VinDr-Mammo) using 200+ classifiers and 4 different LLMs.

Slice Discovery: LADDER significantly outperformed baselines (Domino, Facts) in Precision@10, showing a substantial 50% improvement on medical imaging datasets. It successfully identified ground-truth biases such as "chest tubes" in pneumothorax detection and "water bodies" in bird classification.
Bias Mitigation: In terms of Worst Group Accuracy (WGA), LADDER consistently outperformed state-of-the-art mitigation baselines (GroupDRO, JTT, DFR, etc.).
- On Waterbirds, it achieved a WGA of 91.4% (3.6% improvement over DFR).
- On RSNA-Mammo, it achieved 76.4% (7.3% improvement).
- On VinDr-Mammo, it achieved 82.5% (21.1% improvement).
- On NIH, it outperformed JTT and DFR by 8.2% and 7.4%, respectively.
Robustness: The method demonstrated consistent performance across different architectures (ResNet, ViT) and pretraining strategies (Supervised, Self-Supervised, CLIP).
Zero-Shot Enhancement: Attributes extracted by LADDER were used to construct prompts that improved the zero-shot accuracy of CLIP-based models, with gains up to +19.05% on the NIH dataset.

5. Significance and Claims

The paper claims that LADDER addresses a critical gap in the field: the need for an automated method to discover and mitigate multiple biases without prior knowledge or expensive annotations.

Generalizability: Unlike prior methods restricted to specific training setups or multimodal models, LADDER works on standard ERM-trained classifiers.
Domain Adaptability: By utilizing LLMs, the framework can reason about complex, domain-specific biases (e.g., radiological findings) that are difficult to capture with fixed attribute sets.
Practicality: The approach is cost-effective (total LLM cost for experiments was ~$28) and eliminates the bottleneck of manual attribute annotation.

The authors conclude that LADDER offers a scalable, generalizable approach to uncovering and addressing systematic biases, potentially enhancing the reliability and fairness of AI models in both general computer vision and critical medical applications. They acknowledge limitations, such as a current dependence on captions for discovery (though they propose instruction-tuned models as a workaround) and the potential for inherent biases in the foundational LLMs used.