💻 computer science

Mitigating Feature Entanglement and Imbalance in Driving Style Classification via Physically Isolated Hierarchical Transformers

This paper proposes the Isolated Hierarchical Transformer (IHT), a novel architecture that mitigates feature entanglement and class imbalance in driving style classification by enforcing physically isolated attention mechanisms and integrating Supervised Contrastive Loss, thereby achieving state-of-the-art performance on real-world CAN-bus data.

Original authors: Bingheng Wu, Bin Cai, Ruyi Tang, Dafei Huang

Published 2026-10-08

📖 3 min read☕ Coffee break read

CC BY 4.0

Original authors: Bingheng Wu, Bin Cai, Ruyi Tang, Dafei Huang

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Every time a car moves, its internal computer system generates a constant stream of data describing exactly how the vehicle is behaving. This information, flowing from sensors that track speed, acceleration, and how hard the driver presses the pedals, forms a detailed profile of the person behind the wheel. For engineers building smarter transportation systems, understanding these profiles is crucial. It allows them to design safety features that react to specific driving habits, create insurance policies that reflect actual risk, and train drivers to be safer. However, turning this raw stream of numbers into a clear understanding of a driver's style is surprisingly difficult. The data is messy, and the patterns are often hidden. More importantly, the data is heavily skewed; most drivers behave in a standard, predictable way, while the drivers who are either extremely cautious or dangerously aggressive are rare. This scarcity makes it hard for computer programs to learn what these rare styles actually look like, often causing them to ignore the very drivers they need to recognize most.

A team of researchers at Chongqing University and SERES Automobile Co., Ltd. has developed a new method to solve these problems, turning a complex classification task into a more reliable system. They started with a dataset of over 4,200 real-world driving segments, each labeled by nine experienced experts who watched the data and decided whether the driver was normal, aggressive, or conservative. The experts found that the "conservative" group was very small, representing only a tiny fraction of the total, which created a significant imbalance. To make sense of this, the researchers built a specialized computer model called the Isolated Hierarchical Transformer. Instead of treating every piece of data as a single, flat list of numbers, this model respects the physical nature of the information. It groups related data together—keeping speed statistics separate from pedal movements and acceleration details—so the computer can understand the distinct physical rules governing each type of movement before combining them to form a complete picture.

The researchers also tackled a common mistake made when trying to fix the imbalance of rare driving styles. Often, engineers try to help the computer learn by artificially creating more examples of the rare styles and then telling the computer to pay extra attention to difficult examples. The team discovered that doing both of these things at once actually confuses the system, creating a chaotic learning environment where the computer struggles to find the right boundaries between different driving styles. They found that a more balanced approach, which carefully adds synthetic examples without over-penalizing the difficult ones, worked much better. To further stabilize the learning process, they added a geometric constraint that keeps the examples of the same driving style clustered tightly together in the computer's memory, preventing the artificial examples from drifting into confusing, impossible combinations.

When they tested their new system against existing methods, the results were clear. The new approach correctly identified driving styles with a high degree of accuracy, significantly outperforming the current best models used for this type of data. Most importantly, it became much better at spotting the rare, conservative drivers, improving the detection rate for this group by nearly forty percent compared to older methods. The system achieved this while processing each driving segment in just 4.61 milliseconds, a speed fast enough to be used in real-time applications. By respecting the physical structure of driving data and avoiding the pitfalls of over-correcting for rare events, the researchers have created a more robust tool for understanding human behavior on the road, offering a path toward safer and more responsive intelligent transportation systems.

Technical Summary: Mitigating Feature Entanglement and Imbalance in Driving Style Classification via Physically Isolated Hierarchical Transformers

Problem Statement

Driving Style Classification (DSC) from in-vehicle CAN-bus data is critical for Intelligent Transportation Systems (ITS), including Advanced Driver Assistance Systems (ADAS) and Usage-Based Insurance. However, the field faces two fundamental bottlenecks that hinder robust generalization, particularly under the constraints of limited expert-annotated data and severe class imbalance:

Physics-Agnostic Tokenization: Dominant tabular deep learning models (e.g., FT-Transformer, TabNet) treat kinematic features as independent scalars via flat tokenization. This approach discards physical modality boundaries (e.g., distinguishing between velocity dynamics and pedal operations), forcing the network to implicitly relearn kinematic relationships. Under limited sample sizes ( $N < 5000$ ), this unconstrained global attention risks encoding spurious cross-modal statistical correlations rather than genuine physical dependencies.
The "Over-Correction" Phenomenon: Standard practices for handling class imbalance often combine data-level boundary-focused oversampling (e.g., ADASYN) with algorithm-level boundary-penalizing losses (e.g., Focal Loss). The authors identify that this combination creates conflicting gradient signals. ADASYN concentrates synthetic samples in boundary-overlapping regions, while Focal Loss up-weights gradients from those same noisy samples. This redundancy fragments minority-class decision boundaries rather than reinforcing them, a failure mode termed "over-correction."

Methodology

The paper proposes a unified framework, the Isolated Hierarchical Transformer (IHT), integrated with specific imbalance handling strategies to address both bottlenecks.

1. Architecture: Isolated Hierarchical Transformer (IHT)

The IHT enforces a physics-grounded, two-stage attention mechanism to preserve physical semantics:

Modality-Aware Tokenization: Unlike flat tokenization, IHT assigns independent learnable mapping operators to each feature, projecting heterogeneous scalars into dimensionally aligned, decoupled tokens.
Intra-Modality Self-Attention: Features belonging to the same physical domain (e.g., velocity, acceleration, pedal operations) attend only to each other. This stage purifies local semantic coherence and prevents premature entanglement of disparate physical concepts.
Inter-Modality Cross-Attention: Only after independent encoding does the model employ global cross-attention to capture complex interactions between modalities (e.g., the derivative relationship between pedal input and acceleration).
Design Philosophy: This hierarchical structure acts as a structured information bottleneck, ensuring the model respects kinematic priors while retaining the representational power of global self-attention.

2. Imbalance Handling Strategy

To address the long-tail distribution of driving styles (where "Conservative" and "Aggressive" classes are rare compared to "Normal"), the authors propose a complementary strategy rather than a redundant one:

Sampling: The use of SMOTE (Synthetic Minority Over-sampling Technique) for density restoration in the raw feature space.
Loss Function: Pairing SMOTE with Class-Weighted Cross-Entropy (CW-CCE) rather than Focal Loss to avoid the "over-correction" conflict.
Geometric Regularization: The integration of Supervised Contrastive Loss (SCL). SCL acts as a geometric regularizer in the latent space, enforcing intra-class compactness and inter-class separation. It anchors minority-class manifolds against the noise introduced by linear interpolation in the raw feature space, effectively preventing synthetic samples from fragmenting decision boundaries.

3. Dataset and Feature Engineering

The study utilizes an expert-annotated dataset ( $N=4202$ ) derived from 10 Hz CAN-bus signals.

Annotation: Nine driving experts performed blind evaluations on 40-second signal clips, achieving substantial inter-rater reliability (Fleiss' $\kappa = 0.82$ ).
Feature Engineering: A multi-scale pipeline extracts 32 features:
- Macro Stream: 13 global statistics reduced via PCA to 4 components to prevent linear multicollinearity.
- Micro Stream: 28 local statistics (range, skewness, kurtosis) derived from sliding windows to capture instantaneous maneuvers.
Physical Grouping: Features are logically partitioned into four physical modalities: Macroscopic PCA, Velocity, Acceleration, and Pedal operations.

Key Results

Experiments were conducted using 5-repeat 10-fold stratified cross-validation across 48 model-strategy combinations.

Performance: The complete framework (IHT + SMOTE + CW-CCE + SCL) achieved a Macro F1-score of 0.8415 ( $\pm 0.0452$ ).
Comparative Advantage:
- Outperformed the FT-Transformer (Tabular SOTA) by 1.60% ( $p=0.0415$ ).
- Outperformed TabNet by 2.65% ( $p=0.0185$ ).
- Outperformed the best baseline 1D-CNN by approximately 2.0%.
Minority Class Performance: The F1-score for the minority Conservative class reached 0.7129, a 38% relative improvement over the unoptimized baseline (0.6729).
Ablation Findings:
- Architecture: Removing the hierarchical isolation (using a flat Transformer) dropped performance to 0.8295, confirming the necessity of physical semantic isolation.
- Strategy: The "redundant" combination of ADASYN + Focal Loss resulted in lower performance and higher gradient variance compared to the "complementary" SMOTE + Focal Loss or SMOTE + CW-CCE.
- Feature Disentanglement: IHT reduced feature mutual information redundancy by 52.75% compared to pure baselines and achieved an inter/intra-class Euclidean distance ratio of 5.658, significantly higher than baselines (~2.8).
Efficiency: The model maintains an inference latency of approximately 4.61 ms/sample, demonstrating viability for real-time applications.

Significance and Claims

The paper claims that its primary contribution lies in shifting the paradigm of tabular deep learning for ITS from generic, physics-agnostic architectures to structured feature disentanglement.

Physics-Aware Inductive Bias: The authors assert that the IHT architecture provides a superior inductive bias by explicitly modeling physical modality boundaries. This prevents the network from overfitting to spurious statistical correlations in small datasets, a common failure mode in standard Transformers applied to tabular data.
Quantification of "Over-Correction": The study provides the first empirical characterization and quantification of the "over-correction" phenomenon in vehicular contexts. It demonstrates that stacking multiple Hard Example Mining (HEM) strategies with overlapping mechanisms (boundary synthesis + boundary penalty) is detrimental, whereas complementary strategies (density restoration + geometric regularization) yield synergistic improvements.
Geometric Regularization: The integration of Supervised Contrastive Loss (SCL) is presented as a computationally lightweight alternative to complex generative models for handling synthetic noise. It successfully stabilizes the latent manifold, allowing the model to leverage the benefits of oversampling without succumbing to the noise of physically infeasible synthetic samples.

The authors conclude that while the IHT architecture secures the global baseline performance, the synergistic imbalance strategy is indispensable for the "last mile" of optimizing safety-critical minority events, achieving a robust balance between majority class precision and minority class recall.