🤖 machine learning

A Strong Balanced-Softmax Classifier-Retraining Baseline for Long-Tailed Recognition

This paper introduces BS-cRT, a two-stage retraining baseline that significantly improves few-shot accuracy in long-tailed recognition by freezing a Balanced Softmax-trained backbone and re-optimizing only the classifier on balanced batches, while also analyzing the limitations of boundary-based methods like CBRM.

Original authors: Juan Terven, Diana Margarita Córdova Esparza, Julio Alejandro Romero Gonzalez, Edgar Arturo Chávez Urbiola, Francisco Javier Willars Rodriguez, Juan Bautista Hurtado Ramos, Alfonso Ramirez Pedraza

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Juan Terven, Diana Margarita Córdova Esparza, Julio Alejandro Romero Gonzalez, Edgar Arturo Chávez Urbiola, Francisco Javier Willars Rodriguez, Juan Bautista Hurtado Ramos, Alfonso Ramirez Pedraza

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're running a massive talent show where the judges have to pick winners from thousands of contestants. But here's the twist: 99% of the contestants are from one super-popular city (the "Head" classes), while the other 1% are from tiny, obscure villages (the "Tail" classes).

In a normal talent show, the judges get so used to seeing the popular city's style that they start ignoring the unique talents from the villages. They might give the popular city's singers perfect scores just because they're familiar, while the village singers get crushed, even if they are amazing. This is the problem of Long-Tailed Recognition in AI: the computer gets too good at the common stuff and terrible at the rare stuff.

The First Act: Teaching the Judges Fairly

The paper starts by looking at a method called Balanced Softmax. Think of this as a training camp where the judges are forced to pay attention to the village singers. They learn to appreciate the rare styles, which is great! But the authors asked a simple question: Is the job done?

They found that even after this fair training, the judges (the AI's "classifier") were still a little bit biased. The judges had learned the skills to recognize the villages during the main training, but their final scoring habits were still influenced by the fact that they saw the popular city way more often.

The Big Discovery: Just Retrain the Scorecard

The paper's main finding is a surprisingly simple fix called BS-cRT.

Imagine you've finished the whole talent show training. The judges know the material. Instead of re-teaching them how to sing or dance (retraining the whole AI brain), you just tell them: "Okay, forget the past. Let's do a quick, final practice round where we only show you one singer from the popular city and one from every single village, all mixed up equally."

This is Classifier Retraining. You freeze the brain (the "backbone") so it doesn't forget what it learned, and you only tweak the final scoring sheet (the "classifier") using these perfectly balanced practice rounds.

The Results?
This tiny tweak worked like magic for the rare classes:

On a dataset called CIFAR-100-LT, the AI got +5.15 points better at spotting the rare items.
On CIFAR-10-LT, it jumped +5.83 points.
On the huge ImageNet-LT dataset, it gained +6.92 points.
On Places-LT (a dataset of places), it soared +9.78 points.

The paper is very sure about this: across every single test they ran, this simple "scorecard tweak" consistently made the AI better at the rare stuff without breaking its ability to recognize the common stuff. It's a low-cost, high-reward trick that anyone can use.

The Plot Twist: What Didn't Work

The authors didn't stop there. They wondered if they could do even better by creating fake practice scenarios right at the edge of the judges' decision-making. They called this CBRM.

Imagine trying to help the judges by creating a "fake" contestant that is halfway between a village singer and a popular city singer, right on the line where the judges get confused. The idea was to force the judges to practice on these tricky "boundary" cases.

The paper explicitly rules this out.
They tried this, and it failed. In fact, it made things worse!

When they used the "hardest" fake examples (the ones most likely to be confused with the popular city), the AI's performance on the rare villages dropped by about 4 points.
Why? Because in a long-tailed world, the "hardest" thing a village singer looks like is actually a popular city singer. So, by trying to practice on these hard boundaries, the judges just got pulled back toward the popular city's style. The fake practice sessions were anchored in the wrong place.

The authors are confident this failure isn't just a glitch; it's a fundamental problem with how these "hardest-negative" probes work in imbalanced data. They suggest that if you want to use fake boundary examples, you have to be very careful not to let the popular city hijack the practice session.

The Takeaway

So, what's the lesson for a curious teenager?

Don't overcomplicate it: Sometimes, the best way to fix an AI that ignores rare things isn't to build a giant new brain, but just to give the final decision-maker a quick, fair practice session.
Be careful with "hard" examples: Trying to train on the most difficult, confusing cases can backfire if those cases are actually just the popular stuff in disguise.
The numbers speak: This simple method (BS-cRT) consistently added between 5 and 10 points of accuracy to the rare classes across different datasets. It's a solid, reliable baseline that researchers should use before trying anything fancy.

The paper doesn't claim this solves everything (the AI still has to learn the hard stuff first), but it proves that a simple, balanced re-tuning of the final step is a powerful tool that was hiding in plain sight.

Technical Summary: A Strong Balanced-Softmax Classifier-Retraining Baseline for Long-Tailed Recognition

1. Problem Statement

Long-tailed recognition addresses the challenge where visual categories follow a skewed frequency distribution: a small number of "head" classes dominate the training set, while many "tail" classes appear infrequently. Standard empirical risk minimization causes models to allocate excessive decision space to head classes, resulting in high overall accuracy but poor performance on rare, often critical, categories.

While existing methods address this via loss re-weighting, margin adjustments, or decoupled training (separating representation learning from classifier learning), a specific gap remains: how much residual tail error persists after training a representation with a strong, prior-aware loss like Balanced Softmax? Furthermore, can this error be reduced by retraining only the classifier under balanced exposure, and can synthetic boundary supervision (e.g., probes near decision boundaries) further improve tail performance?

2. Methodology

The paper proposes BS-cRT (Balanced Softmax Classifier Retraining), a minimal two-stage procedure, and evaluates CBRM (Counterfactual Boundary Risk Minimization) as a diagnostic extension.

2.1 BS-cRT: The Core Baseline

BS-cRT isolates the effect of classifier retraining from representation learning.

Stage 1 (Representation Learning): A backbone network ( $f_\theta$ ) and a cosine classifier are trained end-to-end using the Balanced Softmax loss on the original imbalanced data stream. This loss incorporates the empirical class prior ( $\pi_c = n_c / \sum n_{c'}$ ) into the softmax denominator to correct for label shift.
Stage 2 (Classifier Retraining): The backbone is frozen. Only the cosine classifier weights are retrained.
- Sampling: Training uses balanced episodic batches. Each batch samples $P$ classes uniformly without replacement, with $K$ samples per class.
- Loss: The Balanced Softmax loss is retained, using the original empirical training priors ( $\pi_c$ ), not a uniform prior. This ensures the objective remains prior-aware while the exposure to classes is balanced.
- Inference: Predictions use raw cosine logits ( $\ell_c(z) = s \cdot \hat{w}_c^\top z$ ) without adding the log-prior term used during training.

2.2 CBRM: Diagnostic Boundary Supervision

CBRM is evaluated to test if synthetic probes near decision boundaries can improve tail performance beyond BS-cRT.

Mechanism: It maintains multi-prototype class memories. For a sample, it identifies the hardest negative class, constructs a feature-space probe between the positive and negative prototypes, and thickens this probe with orthogonal noise.
Correction: The authors identified a critical failure mode in the original CBRM implementation: a margin/scale mismatch. The original method applied a cosine-margin target to scaled logits, causing gradients roughly 30 $\times$ larger than the Balanced Softmax anchor. The corrected version computes violations directly in cosine-margin space.
Failure Mode Analysis: Even with the corrected loss, CBRM using hardest-negative mining fails to improve over BS-cRT. The analysis suggests that for tail samples, the "hardest negative" is often a head class. Consequently, synthetic probes become head-class anchored, pulling tail classifier directions back toward head-class regions, effectively undoing the benefits of balanced retraining.

3. Key Contributions

Matched-Checkpoint Evaluation: The paper provides a rigorous evaluation of classifier retraining after Balanced Softmax, isolating the effect of the second-stage update from representation learning by using identical Stage 1 checkpoints.
Demonstration of Residual Bias: It demonstrates that even with a strong prior-aware loss in Stage 1, the classifier directions remain biased by the imbalanced training stream. Retraining the classifier under balanced exposure consistently recovers tail accuracy.
CBRM Diagnostics: The paper identifies and fixes a numerical instability in boundary-probe methods (margin/scale mismatch) and reveals a geometric failure mode where hardest-negative probes in long-tailed settings reinforce head-class bias rather than alleviating it.

4. Experimental Results

The method was evaluated on CIFAR-100-LT, CIFAR-10-LT, ImageNet-LT, and Places-LT across imbalance factors (IF) of 50, 100, and 200.

4.1 Performance Gains

BS-cRT consistently improves Few-shot (Tail) accuracy over the matched Balanced Softmax checkpoint:

CIFAR-100-LT (IF=100): Few-shot accuracy increased by +5.15 points (19.91% $\to$ 25.06%), with Top-1 accuracy preserved/improved (40.73% $\to$ 41.07%).
CIFAR-10-LT (IF=100): Few-shot accuracy increased by +5.83 points (64.34% $\to$ 70.17%), with Top-1 improving by +2.17 points.
ImageNet-LT: Few-shot accuracy increased by +6.92 points (38.21% $\to$ 45.13%), though this came with a Top-1 trade-off (-1.64 points).
Places-LT: Few-shot accuracy increased by +9.78 points (18.61% $\to$ 28.39%), with Top-1 improving by +0.74 points.

4.2 CBRM Results

Original CBRM: Caused training instability (loss spikes) and accuracy collapse due to the margin/scale mismatch.
Corrected CBRM (Hardest Negative): Stabilized training but reduced Few-shot accuracy by approximately 4 points compared to BS-cRT on CIFAR-100-LT (25.06% $\to$ 20.70%).
Random Negative: A diagnostic variant using random negative sampling did not show the same tail degradation, supporting the hypothesis that hardest-negative mining causes head-class anchoring.

5. Significance and Claims

The paper positions BS-cRT not as a replacement for complex representation-learning methods, but as a necessary, low-complexity baseline for the field.

Methodological Implication: The authors argue that any long-tailed method claiming tail improvements (via prototypes, contrastive learning, or synthetic data) must first be compared against a matched Balanced-Softmax classifier-retraining baseline. Without this control, gains attributed to complex mechanisms may simply be the result of re-estimating the final classifier under balanced exposure.
Practical Utility: BS-cRT offers a deployment-friendly way to improve tail accuracy without retraining the backbone or adding inference-time modules (latency/memory overhead). It is particularly relevant for scenarios where the backbone is fixed or expensive to retrain.
Boundary Supervision Insight: The paper concludes that synthetic boundary supervision is not automatically beneficial. In long-tailed settings, the selection of negative anchors is critical; using "hardest" negatives often inadvertently reinforces the very bias (head-class dominance) that retraining aims to correct.

In summary, the paper asserts that classifier retraining is a powerful, often overlooked lever for long-tailed recognition, and that BS-cRT serves as the standard reference point for evaluating more elaborate interventions.