💻 computer science

Contextual Token Rotation Ensembles for Tabular Learning

This paper introduces Contextual Token Rotation Ensembles (CTRE), a novel tabular learning framework that enhances ensemble diversity and robustness by integrating Transformer-based contextual token representations with data-adaptive, interaction-aware feature rotations, outperforming classical methods particularly on high-dimensional and imbalanced datasets.

Original authors: Shenghan Gao, Spiridon Penev, Libo Li

Published 2026-08-27

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Shenghan Gao, Spiridon Penev, Libo Li

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

In the world of data science, some problems are solved by looking at pictures or listening to sounds, but many of the most critical decisions in finance, medicine, and engineering rely on tables of numbers. These tables, known as tabular data, are a mix of different types of information: some columns contain continuous measurements like temperature or income, while others hold categories like color or job title. The challenge for computers is that these different types of data do not naturally fit together. A computer struggles to understand how a specific numerical value relates to a specific category unless it is taught to see the hidden connections between them. For decades, researchers have tried to build better ways to read these tables, often by combining many simple prediction models into a single, smarter team. This approach, called ensemble learning, works by ensuring that each member of the team looks at the data in a slightly different way, so that their individual mistakes cancel each other out. However, a long-standing method for creating this diversity has relied on a somewhat haphazard strategy: randomly chopping the columns of data into groups and shuffling them. While this creates variety, it often ignores the fact that some columns are naturally linked, while others are completely unrelated.

A team of researchers at the University of New South Wales has developed a new method called Contextual Token Rotation Ensembles, or CTRE, which replaces this random shuffling with a more intelligent, context-aware process. Instead of treating every column of data as an isolated fact, their system first learns how the columns talk to one another. Imagine a room full of people where everyone is speaking at once; a traditional method might just pick a random group of people to listen to, hoping to catch a useful conversation. The new method, however, first listens to the entire room to understand who is actually talking to whom, and then forms groups based on those real conversations. The researchers achieved this by using a type of artificial intelligence known as a Transformer, which is famous for understanding language. They adapted this technology to look at the columns of a data table as if they were words in a sentence. By training the system to guess missing pieces of data based on the surrounding columns, the computer learns a map of which features depend on each other.

Once this map of relationships is learned, the system uses it to build its team of prediction models. In the old random method, two columns that are deeply connected might be separated into different groups, while two unrelated columns might be forced together. In the new CTRE approach, the system uses the learned map to guide the grouping process. It makes it more likely that columns with strong connections will be placed in the same group, but it still keeps a element of randomness to ensure the team remains diverse. Within each of these smarter groups, the system performs a mathematical transformation that highlights the most important patterns while filtering out the noise. Crucially, the researchers added a step to ensure that the information learned in one group does not accidentally leak into another. They re-process the data for each group in isolation, so that the final prediction for one group depends only on the specific variables assigned to it, preserving the unique perspective of each team member.

The results of this new approach were tested on a wide variety of real-world datasets, ranging from medical records to housing prices and industrial sensor readings. On complex, high-dimensional problems where there are hundreds of different features and many different types of data mixed together, the new method outperformed the best existing techniques. For example, on a dataset involving CT scan slices with 384 different features, the new method reduced the prediction error significantly more than previous top performers. It also showed remarkable strength in handling severely unbalanced data, such as a dataset where only a tiny fraction of entries represent a rare failure or a specific disease. In these difficult cases, older models often ignored the rare events entirely, but the new method successfully identified the subtle patterns associated with them. However, the researchers found that this advanced method is not a universal cure-all. On simpler datasets with fewer features or very uniform data, the extra complexity of learning the relationships between columns sometimes introduced more noise than value, and simpler models performed just as well or better.

The study suggests that the future of tabular learning lies in methods that respect the underlying structure of the data rather than treating it as a random collection of numbers. By teaching the computer to understand how features interact before it tries to make a prediction, the researchers have created a tool that is particularly powerful for the messy, complex, and heterogeneous data found in the real world. The work does not claim to have solved every data problem, but it demonstrates that when data is rich and complex, a method that listens to the connections between variables can build a much more accurate and reliable team of predictors. This shift from random grouping to guided, context-aware grouping represents a meaningful step forward in how machines learn from the structured tables that drive so much of our modern decision-making.

Technical Summary: Contextual Token Rotation Ensembles for Tabular Learning

1. Problem Statement

Tabular learning requires models capable of capturing complex dependencies among heterogeneous features while remaining robust to noise and redundancy. Classical ensemble methods, particularly Rotation Forest, generate diversity by randomly partitioning features into groups and applying Principal Component Analysis (PCA) within each group. However, this approach has two critical limitations:

Ignorance of Dependence Structure: Random partitions may separate features that carry complementary information about the same latent interaction or group unrelated, noisy features together.
Linear Limitations: Applying PCA directly to raw variables only exploits linear covariance structures, failing to capture higher-order or non-linear interactions.

While Transformer-based methods (e.g., TabTransformer, FT-Transformer) excel at learning feature interactions via self-attention, they typically use the Transformer as the final predictor. This paper argues that the Transformer's strength lies in learning the structure of feature interactions, which can be better leveraged to redesign the feature-space transformations of an ensemble rather than serving as the predictor itself.

2. Methodology: Contextual Token Rotation Ensembles (CTRE)

CTRE integrates self-supervised contextual representation learning with selective Rotation Forest-style transformations. The framework operates in four distinct stages:

Stage 1: Feature Tokenization and Contextualization

Tokenization: Numerical and categorical features are mapped into a common continuous token space. Numerical features are standardized and linearly projected; categorical features are one-hot encoded and embedded. A column embedding is added to preserve feature identity.
Contextualization: A Transformer encoder processes the sequence of tokens. Unlike standard predictors, the encoder does not pool tokens into a single sample representation. Instead, it produces a contextual token for each original feature, where the representation of feature $j$ depends on the entire feature set via self-attention.
Pretraining: The tokenizer and encoder are pretrained using masked-feature reconstruction (predicting masked features from the rest) and optional consistency regularization. The target variable is excluded during this phase. The resulting attention matrices capture representation-level affinities between features.

Stage 2: Selective Stochastic Feature Grouping

Instead of random partitioning, CTRE constructs feature groups for each ensemble member using the learned attention structure:

Affinity Matrix: Attention matrices from all layers and heads are aggregated into a symmetric feature-affinity matrix $W$ , measuring the average interaction strength between feature pairs.
Stochastic Selection: For each ensemble member, features are grouped based on a probability distribution that balances:
1. Interaction-Guided Selection: Features with high affinity to the current group are more likely to be selected.
2. Uniform Randomness: A uniform component ensures exploration and prevents deterministic collapse.
3. Repeated-Pairing Penalty: A penalty term discourages feature pairs that have frequently appeared together in previous ensemble members, ensuring diversity across the ensemble.

Stage 3: Group-Wise Contextual Re-Encoding and PCA

Re-Encoding: To prevent "cross-group representation leakage" (where a token in Group A contains information from features in Group B due to global attention), the selected features for a specific group are re-encoded using the same pretrained encoder. This ensures the contextual representation for a group depends only on the features within that group.
Group-Wise PCA: PCA is applied to the re-encoded, vectorized contextual tokens within each group. An optional explained-variance threshold allows for dimensionality reduction by discarding low-variance directions.
Rotation: The PCA scores from all groups are concatenated to form a rotated representation specific to that ensemble member.

Stage 4: Ensemble Prediction

Each rotated representation is fed into an independently trained Multilayer Perceptron (MLP). Final predictions are obtained by uniform averaging of the member predictions.

3. Key Contributions

The paper identifies two primary methodological contributions:

Selective Stochastic Grouping: CTRE replaces arbitrary random grouping with a mechanism that uses learned attention affinities to make strongly interacting features more likely to appear in the same group. Crucially, it retains a stochastic component and a repeated-pairing penalty to maintain ensemble diversity, avoiding the pitfalls of a single deterministic partition.
Group-Wise Contextual Re-Encoding: The method addresses the issue of cross-group information leakage. By re-encoding features within a group after partitioning, the method ensures that the transformed subspace for a group is structurally independent of features outside that group. This preserves the structural meaning of the partition and reduces artificial similarity between ensemble members.

4. Theoretical Analysis

The authors provide a theoretical analysis grounded in the strength–diversity decomposition of ensemble learning ( $R_{ens} = R_{base} - D_{amb}$ ):

Strength: Theoretical results (Propositions 2 and 3) suggest that contextual aggregation and coherent grouping improve the recovery of shared latent factors and concentrate signal into leading PCA directions, thereby reducing the base-learner risk ( $R_{base}$ ).
Diversity: The analysis (Propositions 6–8) demonstrates that the stochastic grouping, reuse penalties, and group-wise re-encoding preserve the non-degenerate support of the partition distribution and eliminate cross-group information flow, thereby maintaining predictive diversity ( $D_{amb}$ ).

5. Experimental Results

CTRE was evaluated on five regression and six classification benchmarks, including high-dimensional, heterogeneous, and severely imbalanced datasets.

Performance: CTRE achieved state-of-the-art or competitive results on high-dimensional and heterogeneous datasets.
- Regression: It achieved the lowest RMSE on CT Slice Localization, BlogFeedback, and Ames Housing, outperforming strong baselines like XGBoost and TabM.
- Classification: It achieved the best balanced accuracy on ISOLET and HAR.
Imbalanced Data: CTRE demonstrated significant advantages on severely imbalanced datasets (KDDCup09 Churn and APS Failure). While many baselines collapsed to predicting the majority class (balanced accuracy $\approx 0.50$ ), CTRE achieved significantly higher balanced accuracy (0.6258 and 0.9643, respectively), indicating an ability to learn patterns from rare classes.
Limitations: On low-dimensional or homogeneous datasets (e.g., California Housing, Superconductivity), CTRE underperformed compared to simpler baselines. The authors attribute this to the encoder potentially overfitting or introducing representation noise when feature structure is insufficient relative to model capacity.

Ablation Studies:
Experiments confirmed that the full CTRE pipeline (tokenization, interaction-guided grouping, re-encoding, and ensemble) performs best. Removing group-wise re-encoding or using single learners significantly degraded performance, validating the necessity of the proposed components.

6. Significance and Claims

The paper positions CTRE not as a universal replacement for all tabular methods, but as a specialized framework for problems with sufficiently rich feature interactions, high dimensionality, and heterogeneity.

Paradigm Shift: The work shifts the role of Transformers in tabular learning from being the final predictor to being a structure-learning module that informs the construction of rotation-based ensembles.
Balanced Trade-off: The method explicitly addresses the strength-diversity trade-off by using learned affinities to strengthen base learners while using stochasticity and re-encoding to preserve ensemble diversity.
Modest Scope: The authors modestly claim that CTRE is particularly effective for high-dimensional, heterogeneous, and imbalanced problems but may not be suitable for low-dimensional or homogeneous settings where the cost of representation learning outweighs the benefits of rotation.

Future work is suggested to focus on data-dependent criteria for method selection, adaptive parameter tuning, and computational efficiency improvements for group-wise re-encoding.