📊 statistics

Large Language Models for Imbalanced Classification: Diversity makes the difference

This paper proposes a novel large language model-based oversampling method that enhances the diversity and realism of synthetic minority samples through a conditional generation strategy, a new permutation-based fine-tuning approach, and interpolation, thereby significantly outperforming existing state-of-the-art techniques in imbalanced classification tasks.

Original authors: Dang Nguyen, Sunil Gupta, Kien Do, Thin Nguyen, Taylor Braund, Alexis Whitton, Svetha Venkatesh

Published 2026-06-09

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Dang Nguyen, Sunil Gupta, Kien Do, Thin Nguyen, Taylor Braund, Alexis Whitton, Svetha Venkatesh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

The Problem: The "Rare Disease" Classroom

Imagine a teacher trying to teach a class about two types of students: "Regular Students" (who make up 90% of the class) and "Rare Students" (who make up only 10%).

If the teacher only looks at the 90% majority, they will learn everything about the "Regular Students" but will know almost nothing about the "Rare Students." When a test comes, the teacher will likely guess that everyone is a "Regular Student" because that's what they've seen most often. They fail to recognize the unique traits of the minority group.

In data science, this is called Imbalanced Classification. The goal is to teach the computer to recognize the rare group just as well as the common group.

The Old Solution: The "Copy-Paste" Mistake

To fix this, data scientists usually try to create more examples of the "Rare Students" so the class is balanced.

The Old Way (SMOTE): Imagine taking two real "Rare Students," measuring them, and drawing a new student right in the middle of them. It's like making a photocopy of a photocopy.
The Problem: If the data has categories (like "Job Title" or "Education Level"), you can't just draw a line between "Doctor" and "Teacher" to get a new job. You have to turn those words into numbers first, which often loses important details. It's like trying to describe a painting using only a spreadsheet.

The New Idea: The "AI Storyteller"

Recently, scientists started using Large Language Models (LLMs)—the same kind of AI that writes essays or chats with you—to generate these missing "Rare Students." Instead of turning words into numbers, the AI reads the data as a story (e.g., "John is a Doctor, works 40 hours, and earns $200k").

However, the paper found a major flaw in how these AI storytellers were being used:
They were too repetitive. If you asked the AI, "Tell me a story about a Rare Student," and it gave you the exact same story 1,000 times, it wouldn't help the teacher learn anything new. The AI was just repeating the same few patterns, creating a "hall of mirrors" instead of a diverse classroom.

The Solution: ImbLLM (The "Diverse Storyteller")

The authors propose a new method called ImbLLM. They fixed the AI's storytelling habits with three clever tricks to ensure the new "students" are diverse and realistic.

1. The "Specific Prompt" Trick (Sampling)

Old Way: The AI was asked, "Tell me a story about a Rare Student." It would pick the most likely words and repeat the same story over and over.
ImbLLM Way: The authors tell the AI, "Tell me a story about a Rare Student who also has a specific feature, like a specific job or income."
The Analogy: Instead of asking a chef, "Make me a dessert," and getting the same chocolate cake every time, you say, "Make me a dessert, but this one must have strawberries." Then, next time, you say, "Make me a dessert, but this one must have lemons." This forces the AI to explore different flavors (features) rather than sticking to one safe recipe.

2. The "Front-Row Seat" Trick (Permutation)

Old Way: When training the AI, the researchers would shuffle the order of the story. Sometimes the "Rare Student" label was at the end of the sentence. Because of how these AI models work (they read left-to-right), the AI would forget the "Rare" label by the time it got to the end, losing the connection between the label and the features.
ImbLLM Way: They keep the "Rare Student" label at the very beginning of the sentence, like a title, and only shuffle the rest of the details.
The Analogy: Imagine a detective trying to solve a case. If the clue "The suspect is a Doctor" is buried at the bottom of a long report, the detective might miss it. ImbLLM puts the clue in big, bold letters at the top of the page so the AI never forgets who it is supposed to be describing.

3. The "Practice Run" Trick (Fine-Tuning & Interpolation)

Old Way: The AI was trained on both the common students and the rare students. This confused the AI; it kept trying to make the "Rare Students" look like the "Common Students." Also, some methods tried to check if the new stories were "real" using a weak test, which often failed.
ImbLLM Way: They trained the AI only on the "Rare Students" (and some mathematically blended versions of them).
The Analogy: Instead of teaching a basketball player by showing them both NBA stars and toddlers, you only show them NBA stars. You want them to learn the specific moves of the pros.
The "Interpolation" Bonus: Since there are very few "Rare Students" to learn from, the AI might not know all the possible variations. The authors created "practice students" by taking two real rare students and mathematically blending their continuous traits (like age or salary) to create a new, slightly different student. This fills in the gaps, ensuring the AI learns the full range of possibilities, not just the ones it saw in the training data.

The Results: A Better Classroom

The authors tested this new method on 10 different real-world datasets (like medical records or credit card data).

Performance: ImbLLM beat 8 other top methods. It was the best in 5 cases and second-best in 3 others.
Quality: The fake "Rare Students" created by ImbLLM were not only realistic (they looked like real data) but also diverse (they covered many different types of rare scenarios).
Why it matters: Because the AI learned from a diverse set of examples, the final classifier (the "teacher") became much better at spotting the rare cases it had previously ignored.

Summary

The paper argues that simply using AI to generate data isn't enough; you have to teach the AI how to be creative and focused. By forcing the AI to look at specific features, keeping the main goal in clear view, and training it exclusively on the rare examples, they created a method that solves the "imbalanced data" problem much better than previous attempts.

Technical Summary: Large Language Models for Imbalanced Classification: Diversity makes the difference

Problem Statement

Imbalanced classification remains a critical challenge in machine learning, particularly when dealing with tabular data where minority classes (e.g., rare diseases) have significantly fewer samples than majority classes. Traditional oversampling methods, such as SMOTE, address this by generating synthetic minority samples. However, these approaches often require converting categorical variables into numerical vectors (via ordinal or one-hot encoding), leading to information loss and the introduction of artifacts.

While recent Large Language Model (LLM)-based methods have emerged to handle tabular data as text—preserving variable contexts and avoiding numerical conversion—they suffer from a different limitation: limited diversity. Existing LLM-based approaches typically generate synthetic minority samples that are too similar to one another. This lack of diversity reduces the robustness and generalizability of downstream classifiers. Furthermore, current methods often rely on unreliable verification steps (using external classifiers or the LLM itself trained on imbalanced data) to filter synthetic samples, which can lead to the rejection of valid data.

Methodology: ImbLLM

The authors propose ImbLLM, a novel LLM-based oversampling framework designed to enhance the diversity and quality of synthetic minority samples. The method introduces three specific technical improvements over existing baselines:

1. Diversity-Conditioned Sampling (`condition_yx`)

Existing methods construct prompts solely based on the minority label (e.g., "Income is ≥200K") to query the LLM. Due to the softmax function used in token prediction, identical prompts often yield highly similar outputs.

Innovation: ImbLLM constructs prompts conditioned on both the minority label and a randomly sampled feature.
Mechanism: For each generation, the system samples a feature $X_i$ and its value $v_i$ from the minority distribution, forming a prompt like "Income is ≥200K, Edu is Bachelor." This variation in conditioning inputs forces the LLM to explore different regions of the feature space, generating more diverse synthetic samples.

2. Feature Permutation with Fixed Label (`fix_y`)

Standard LLM-based methods permute both features ( $X$ ) and the target variable ( $Y$ ) during fine-tuning to prevent positional bias. However, in decoder-only LLMs, the attention mechanism is masked such that a token can only attend to preceding tokens.

Problem: If the minority label $Y$ is shuffled to the middle or end of a sequence, it cannot attend to features preceding it, limiting the model's ability to generate diverse features conditioned on that label.
Innovation: ImbLLM fixes the minority label ( $Y$ ) at the beginning of the sentence and only permutes the features ( $X$ ).
Mechanism: By placing $Y$ first (e.g., "Income is ≥200K, Edu is Bachelor..."), the label can attend to all subsequent features via the attention mechanism, ensuring the generated features are fully conditioned on the target class.

3. Fine-Tuning with Interpolation (`minor+interpolate`)

Existing methods fine-tune LLMs on both majority and minority data, then attempt to verify synthetic samples to ensure they belong to the minority class. This verification is often unreliable because the verifier is trained on imbalanced data.

Innovation: ImbLLM fine-tunes the LLM only on minority samples and interpolated samples, completely removing the majority class from the fine-tuning phase.
Mechanism: To address the scarcity of minority data (which may lack sufficient coverage of continuous variable domains), the authors introduce an interpolation step. They generate synthetic continuous values by interpolating between existing minority samples ( $x'_i = x^c_i + \epsilon(x^c_j - x^c_i)$ ). The LLM is then fine-tuned on the union of original minority samples and these interpolated samples. This eliminates the need for a post-generation verification step and prevents the model from being biased toward the majority class.

Key Contributions

ImbLLM Framework: A novel oversampling method integrating three key improvements (sampling, permutation, and fine-tuning) to generate diverse and robust minority samples.
Theoretical Analysis: An entropy-based analysis proving that the proposed generation process mathematically encourages diversity in the output.
Comprehensive Evaluation: Extensive experiments on 10 real-world tabular datasets comparing ImbLLM against eight state-of-the-art baselines (including SMOTE, CTGAN, and other LLM-based methods like Great, TapTap, and Pred-LLM).
Quality and Diversity Metrics: Demonstration that generated samples are not only more effective for classification but also superior in "Close Probability" (similarity to real data) and "Coverage" (diversity of the generated manifold).

Experimental Results

The authors evaluated ImbLLM on 10 datasets (e.g., Adult, German, Credit Card) using F1-score and AUC as primary metrics for downstream classification performance.

Classification Performance: ImbLLM achieved the best F1-score on 5 out of 10 datasets and the second-best on 3 others. On average, it outperformed the runner-up (CTGAN) by approximately 3% and the standard imbalanced classifier (without oversampling) by 8%. Notably, ImbLLM's performance approached that of a classifier trained on the original, balanced dataset, despite using only 20% of the original minority samples for training.
Diversity and Quality:
- Close Probability: ImbLLM scored 0.8011 compared to CTGAN's 0.7761, indicating synthetic samples are closer to real minority samples.
- Coverage: ImbLLM scored 0.8933 compared to CTGAN's 0.8482, indicating better coverage of the real data manifold.
Ablation Studies: The study confirmed that each component (diverse sampling, fixed-label permutation, and interpolation) contributes significantly to performance. Removing the interpolation step resulted in lower performance, though still superior to baselines.
Robustness: Unlike TapTap and Pred-LLM, which rely on verification steps that fail when the imbalanced dataset is too skewed to train a reliable verifier, ImbLLM's approach of excluding majority data during fine-tuning proved more robust across varying imbalance ratios.

Significance and Claims

The paper claims that diversity is the critical factor distinguishing successful LLM-based oversampling from failure. While previous LLM methods focused on the mechanics of generation or verification, ImbLLM demonstrates that optimizing the diversity of the generation process—through specific prompt engineering, attention-aware permutation, and interpolation-based fine-tuning—directly translates to superior downstream classification performance.

The authors assert that their method provides a more reliable alternative to verification-heavy approaches, as it inherently generates high-quality minority data without needing to filter out "bad" samples using potentially biased classifiers. The results suggest that ImbLLM can effectively bridge the performance gap between severely imbalanced datasets and their balanced counterparts, making it a viable solution for real-world applications involving rare events in tabular data.