Technical Summary: Divide and Contrast (Di-COT)

Problem Statement

Self-supervised learning (SSL) for time-series representation aims to reduce reliance on labeled data while maintaining strong downstream performance. However, existing contrastive learning approaches face significant limitations:

1. Computational Inefficiency: Many methods require multiple passes through the encoder (for anchors, positives, and negatives) or rely on expensive operations like Dynamic Time Warping (DTW) for similarity computation.
2. Data Augmentation Bias: Approaches that rely heavily on data augmentation (e.g., masking, time-domain/frequency-domain transformations) can introduce inductive biases that distort the learned feature embeddings, hindering generalizability.
3. Temporal Assumption Failures: Recent efficient methods (e.g., CaTT) that contrast temporally adjacent timesteps assume that temporal proximity implies semantic similarity. This assumption often fails in datasets with rapid event transitions (common in UCR and UEA benchmarks), leading to "false positives" where semantically distinct but temporally close timesteps are incorrectly treated as positive pairs.
4. Scalability Issues: Loss computations in some methods scale quadratically with both batch size and sequence length, making them prohibitive for long time-series windows.

Methodology: Divide and Contrast (Di-COT)

Di-COT is an unsupervised framework designed to learn robust temporal features without data augmentation or multiple encoder passes. It reframes temporal contrastive learning as a dense classification task within the representation space.

Core Mechanism

1. Stochastic Sub-Block Partitioning: Instead of processing the entire sequence or individual timesteps, Di-COT stochastically partitions each time-series window into $k$ overlapping sub-blocks per iteration. The number of sub-blocks $k$ is sampled from a uniform distribution $U\{k_{min}, \dots, k_{max}\}$.

   • The sub-blocks have length $L$ and stride $s$, controlled by an overlap ratio $\rho \in (0, 1)$.
   • This stochasticity exposes the encoder to multiple temporal granularities, implicitly encouraging robustness across short and long-range contexts without explicit hierarchical designs.

2. Sub-Block Encoding: Each sub-block is passed through a shared temporal encoder $f_\theta$ and pooled along the temporal dimension to produce a single embedding vector.

3. Contrastive Objective (Next-Sub-Block Prediction):

   • Positive Pairs: Temporally adjacent sub-blocks within the same instance are treated as positive pairs. Specifically, for a sub-block $j$, the preceding sub-block $j-1$ is the target.
   • Negative Pairs: All other sub-blocks in the same sequence serve as negatives.
   • Formulation: The task is formulated as a multi-class classification problem. For each sub-block $j$, the model predicts the index of its preceding neighbor. The loss is a temperature-scaled cross-entropy loss (similar to InfoNCE but applied to sub-block indices):
     $$\mathcal{L}_{CE} = -\frac{1}{Bk} \sum_{i=1}^{B} \sum_{j=0}^{k-1} \log \frac{\exp(S^{(i)}_{j, p^*(j)} / \tau)}{\sum_{p=0}^{k-1} \exp(S^{(i)}_{j, p} / \tau)}$$
     where $S^{(i)}_{j,p}$ is the similarity between sub-block $j$ and $p$, and $p^*(j)$ is the target index ($j-1$ for $j>0$, and $0$ for the first sub-block).

Efficiency and Scalability

• Complexity: The loss computation depends on the batch size $B$ and the number of sub-blocks $k$, rather than the sequence length $T$. Since $k \ll T$ by design, the time complexity is $O(Bk^2d)$ and memory complexity is $O(Bk^2)$. This makes the method independent of sequence length, allowing efficient training on very long sequences.
• No Augmentation/Multiple Passes: By contrasting internal substructures, Di-COT avoids the need for data augmentation and eliminates the need for multiple encoder passes required by methods that treat different augmented views as positives.

Key Contributions

1. Augmentation-Free Framework: Di-COT eliminates the need for data augmentation, timestep masking, and multiple encoder passes, reducing computational overhead and preventing representation distortion.
2. Dynamic Sub-Block Contrasting: The method introduces contrasting within individual instances using stochastic sub-blocks. This prevents trivial timestep-level discrimination (which fails during rapid transitions) and promotes diverse, semantically meaningful representations.
3. Reformulation as Classification: Temporal contrastive learning is reformulated as a cross-entropy classification task. This provides dense supervision (utilizing every sub-block as a signal) while retaining the properties of temperature-scaled InfoNCE.
4. State-of-the-Art Performance: The framework achieves state-of-the-art results across classification, clustering, kNN, and cross-dataset transfer tasks while substantially reducing training time.

Experimental Results

The authors evaluated Di-COT on six large-scale real-world datasets (PAMAP2, WISDM2, HARTH, SLEEP, ECG, SKODA) and the UCR/UEA benchmarks.

• Linear Evaluation: Di-COT achieved the highest average accuracy among SSL methods on the large-scale datasets, outperforming baselines like TS2Vec, TF-C, and CaTT. It matched the performance of fully supervised end-to-end models on several datasets while requiring significantly less training time.
• Low-Label Regime: In a 1% labeled data setting, Di-COT outperformed all other SSL methods and the fully supervised baseline trained end-to-end on the same limited data, demonstrating superior label efficiency.
• Non-Parametric kNN: Di-COT showed the best performance in 1-nearest neighbor classification, indicating that the learned embeddings have a well-structured local neighborhood where semantically similar samples are closely mapped.
• Clustering: Di-COT achieved the highest Normalized Mutual Information (NMI) and Adjusted Rand Index (ARI), suggesting it learns globally coherent clusters that align with ground-truth classes.
• Cross-Dataset Transfer: In transfer learning experiments (e.g., HARTH $\to$ PAMAP2), Di-COT demonstrated stronger domain generalization compared to baselines, indicating robustness to distribution shifts.
• Efficiency: Di-COT consistently required the lowest training time among deep learning-based approaches. For instance, on large-scale datasets, it trained in approximately 2.88 hours compared to 3.47–11.48 hours for other methods.

Significance and Claims

The paper claims that Di-COT addresses the fundamental trade-off in existing time-series contrastive learning methods: the tension between exploiting temporal proximity for efficiency and maintaining semantic consistency for generalization.

• Robustness to Temporal Dynamics: By contrasting informative substructures rather than individual timesteps, Di-COT avoids the "false positive" problem inherent in methods that assume all adjacent timesteps are semantically similar. This makes it particularly effective for datasets with frequent event changes.
• Scalability: The independence of loss computation from sequence length allows Di-COT to scale efficiently to long time-series, a limitation for methods like CaTT where memory and compute costs scale quadratically with sequence length.
• Practical Applicability: The framework is presented as a practical solution for scenarios with scarce annotations, offering a favorable accuracy-compute trade-off. The authors note that while the method excels in classification, clustering, and retrieval, its contrastive nature (focused on discriminative structure) may be orthogonal to forecasting tasks requiring fine-grained temporal prediction, which is identified as a limitation and direction for future work.

The authors emphasize that Di-COT learns representations that are locally consistent, globally well-organized, and transferable across domains, establishing a new state-of-the-art for unsupervised time-series representation learning.