💻 computer science

Automatic Discovery of Intra-Class Sub-Structure for Supervised Tabular Classification: Offline Clustering vs. Joint Sub-Center Training

This rigorous empirical study demonstrates that while conventional offline clustering of penultimate features to discover intra-class sub-structure is unreliable and often degrades tabular classification performance, a joint end-to-end sub-center training approach effectively mitigates these risks, though the authors conclude that no robust heuristic currently exists to predict when such sub-structure discovery is beneficial.

Original authors: Seyed Ali Zaribaf, Mohammad Roustaei

Published 2026-07-30

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Seyed Ali Zaribaf, Mohammad Roustaei

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize different types of furniture. You show it pictures of chairs, tables, and sofas. But here's the catch: the label "chair" is a bit lazy. A wooden dining chair, a fluffy armchair, and a high-tech gaming chair are all just "chairs" to your robot, even though they look and feel very different. In the world of machine learning, this is called tabular classification. It's the art of teaching computers to sort data into buckets based on rules we give them. Usually, we assume each bucket (like "chair") holds one kind of thing. But in the real world, buckets are often messy mixtures.

The big question researchers have been asking is: Can we teach the computer to secretly figure out these hidden sub-types on its own? If the robot could realize, "Oh, this 'chair' is actually a 'wooden chair' and that one is an 'armchair'," maybe it would get better at sorting them. This idea isn't new; it's like having a detective who looks at a crowd of people labeled "students" and realizes there are actually "math students," "art students," and "gym students" mixed in. If the detective can spot these groups, they might solve the mystery faster. But there's a risk: what if the detective starts seeing patterns that aren't really there, like thinking every student with a red shirt is a "math student" when they aren't? That's the danger of trying to find hidden structure where none exists.

This paper is a rigorous reality check on that idea. The authors, Seyed Ali Zaribaf and Mohammad Roustaei, decided to test two different ways to help a computer find these hidden "sub-classes" in messy data. They didn't just guess; they ran a massive experiment across ten different datasets, testing their theories five times each to be sure.

First, they tried the "obvious" method, which they call the offline clustering pipeline. Imagine you train a robot to recognize chairs, then you pause, take a snapshot of what the robot has learned, and ask a separate tool (called k-means) to group the "chairs" into smaller piles. Then, you tell the robot, "Hey, these aren't just chairs; they are 'Group A chairs' and 'Group B chairs'!" and you make the robot relearn everything from scratch with these new labels. The result? This approach was a disaster. Out of ten different datasets, it only helped on two, and on the other eight, it actually made the robot worse at its job. In fact, on some datasets, the robot's accuracy dropped by as much as 2.8 percentage points. The authors found that this method is unreliable because it often forces the robot to invent fake patterns where there are none, confusing it rather than helping it.

Next, they tried a smarter, more integrated approach called joint sub-center training. Instead of pausing to re-label things, they gave the robot a special "super-layer" at the very end of its brain. This layer allows the robot to say, "I think this is a chair, but I'm also considering that it might be a 'wooden chair' or an 'armchair' all at the same time," and then it combines those thoughts to make a final decision. They trained this whole system together, from start to finish. The result was a huge improvement in safety. This method never made the robot significantly worse. On the datasets where the first method failed miserably, this new method actually helped, improving accuracy by small but real amounts (like 0.41 percentage points on the Yeast dataset).

Why did the second method work while the first failed? The authors discovered a fascinating phenomenon they call "expert collapse." Even though they gave the robot a budget to create up to 10 different sub-groups for every class, the robot naturally decided it only needed 1 or 2 of them. It was like giving a chef a kitchen with 10 burners, but the chef only turned on 2 because that's all they needed to cook the meal perfectly. The robot automatically figured out which sub-groups were real and ignored the rest, preventing it from making up fake patterns.

The paper also offers a few other interesting findings. They proposed a simple "rule of thumb" to guess if finding sub-groups would help: if a Random Forest (a different type of smart algorithm) beats your robot by a lot, maybe there's hidden structure to find. However, they admit this rule is only a weak hint; it worked correctly on 13 out of 17 datasets, which is only slightly better than just guessing "no" every time. They also uncovered a sneaky mistake they made early on: they had trained one version of the robot poorly, which made it look like the robot's internal "thoughts" were better than its final answer. Once they fixed the training, the robot's final answer was actually just as good as its internal thoughts, proving that the "magic" wasn't in the thoughts themselves, but in how well the robot was trained.

In the end, this paper tells us that while trying to find hidden sub-groups in data is a great idea, the "stop-and-relabel" method is too risky and often backfires. Instead, building a flexible system that learns these sub-groups while it learns the main task is much safer. It's a reminder that in science, sometimes the most obvious path is a dead end, and the best solution is one that adapts and self-corrects as it goes.

Technical Summary: Automatic Discovery of Intra-Class Sub-Structure for Supervised Tabular Classification

Problem Statement
In supervised tabular classification, class labels are often coarser than the underlying data-generating process. A single label (e.g., "chair" or "Type A failure") may encompass multiple distinct sub-types or failure modes. The central research question is whether automatically discovering and exploiting this hidden intra-class sub-structure can improve classification accuracy without relying on complex foundation models or unsupervised pretraining. The paper investigates two primary approaches to this problem: a two-stage offline clustering pipeline and a joint end-to-end sub-center training mechanism.

Methodology
The study employs a standard two-hidden-layer Multi-Layer Perceptron (MLP) with an expansion (overcomplete) penultimate layer as the base classifier. The authors rigorously compare two strategies for modeling intra-class heterogeneity across ten diverse tabular datasets (including Letter, MNIST, HAR, and Yeast), using 5 independent random seeds per dataset and paired significance testing.

Offline Clustering-Relabel Pipeline (Two-Stage):
- Step 1: Train a base network on coarse labels.
- Step 2: Extract penultimate features and apply $k$ -means clustering independently within each class to generate sub-labels.
- Step 3: Retrain a fresh network from scratch using these new sub-labels.
- Step 4: At inference, predict a sub-label and map it back to the parent class.
- The authors perform a full sweep over $k \in \{2, 3, 5, 10\}$ to determine if specific hyperparameters drive success or failure.
Joint Sub-Center Head (End-to-End):
- Replaces the standard output layer with a sub-center layer having $C \times k$ logits (where $C$ is the number of classes and $k$ is the number of sub-centers).
- The model is trained end-to-end using a marginalized softmax likelihood, where the probability of a class is the sum of probabilities of its $k$ sub-components.
- No separate clustering stage or relabeling occurs; the network learns the sub-structure implicitly.
Heuristic and Controls:
- RF-Gap Heuristic: A diagnostic tool predicting whether sub-class discovery will help based on the accuracy gap between a Random Forest (RF) and the base MLP.
- Parameter-Matched Control: A non-marginalized variant (mean-pooled logits) to distinguish between the benefits of the mixture-model objective versus simple output-layer width expansion.
- Frozen-Encoder Diagnostic: To determine if gains come from the output layer flexibility or changes in the shared representation.

Key Results

Offline Clustering is Unreliable: The two-stage offline pipeline is empirically unstable. Across ten datasets, it improved accuracy on only 2/10 (marginally) and degraded performance on 8/10. On six datasets, the degradation was statistically significant (up to -2.83 percentage points on Yeast). A sweep over $k$ revealed that degradation worsened monotonically as $k$ increased on 8/10 datasets, suggesting the method is prone to manufacturing spurious partitions that harm coarse-label accuracy.
Joint Sub-Center Training is Safe: Replacing the offline pipeline with the joint sub-center head eliminated the failure mode. The worst degradation observed was -0.03 percentage points (statistically indistinguishable from noise). The joint method improved accuracy on datasets where the offline method failed (e.g., Yeast, Vowel).
Self-Limiting Mechanism (Effective- $k$ ): Despite being allocated a budget of $k=10$ sub-centers, the joint head self-limits to 1–2 active components per class on 9/10 datasets. This "expert collapse" prevents the model from overfitting to noise in classes that lack genuine sub-structure, acting as a form of implicit regularization.
Role of Marginalization vs. Capacity: A parameter-matched control (mean-pooled logits without marginalization) performed statistically indistinguishably from the marginalized joint head on all 10 datasets. This suggests that at the current sample size, the safety margin is attributable to the increased output-layer width rather than the specific marginal-likelihood mechanism, though the joint head remains strictly safer than the offline approach.
RF-Gap Heuristic: The heuristic (predicting "split" if RF accuracy exceeds MLP by >2pp) correctly predicted the direction of improvement on 13/17 datasets (76%). However, this is only marginally better than a trivial "always skip" baseline (70.6%), and the correlation is not statistically significant ( $r=0.37, p=0.15$ ). It serves as a weak, directionally useful rule of thumb rather than a robust predictor.
Training-Budget Artifact: The authors identified a methodological artifact where an earlier implementation using full-batch gradient descent for the end-to-end network (vs. a well-tuned solver for the linear probe) falsely suggested penultimate features were superior. When both were trained with equal mini-batch budgets, the end-to-end output matched or exceeded the probe on 8/10 datasets, correcting a previous misinterpretation.

Significance and Claims
The authors position this work primarily as a rigorous empirical and diagnostic study rather than a proposal of a novel architecture. Their contributions are:

Empirical Characterization: Providing a systematic, statistically rigorous comparison showing that the intuitive offline clustering approach is unreliable for supervised tabular classification, while the joint sub-center approach offers a safe alternative that avoids failure modes through self-regularization.
Diagnostic Insights: Uncovering the "effective- $k$ " phenomenon (self-limiting sub-centers) and the training-budget artifact, which clarifies why certain comparisons in literature may be flawed.
Honest Limitations: The paper explicitly states it does not claim architectural novelty for the sub-center head (which adapts existing mechanisms from ACOL and Sub-center ArcFace) nor novelty in MLP performance against tree ensembles. The value lies in the honest accounting of statistical power, the transparent reporting of null results (e.g., the indistinguishability of marginalized vs. mean-pooled heads), and the cautionary findings regarding offline clustering.

The study concludes that while automatic sub-structure discovery via joint training is a safe mechanism that does not degrade performance, its benefits are modest and often indistinguishable from the flexibility of a wider output layer. The offline clustering approach, conversely, should be treated with caution due to its high risk of significant performance degradation.

Technical Summary: Automatic Discovery of Intra-Class Sub-Structure for Supervised Tabular Classification

More like this