📊 statistics

Beyond Looking Up, Try Looking Around: Harmonizing Global Structure and Local Consistency in Optimal Transport for Short Text Clustering

This paper proposes a novel short text clustering framework that enhances Optimal Transport by integrating an instance-level attention mechanism to capture local semantic consistency, thereby generating reliable pseudo-labels that harmonize neighborhood relationships with global cluster structures to outperform state-of-the-art methods.

Original authors: Zhihao Yao, Yuxuan Gu, Jixuan Yin, Bo Li

Published 2026-07-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Zhihao Yao, Yuxuan Gu, Jixuan Yin, Bo Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to organize a massive, chaotic pile of short notes—like tweets, search queries, or text messages—into neat groups. Maybe you want to sort them by topic: "cats," "sports," or "politics." This is the job of short text clustering.

For a long time, the smartest way to do this involved a mathematical tool called Optimal Transport (OT). Think of OT as a super-efficient delivery service. It looks at every single note (a "sample") and tries to figure out which "warehouse" (a cluster) it belongs to. The goal is to move all the notes to their warehouses with the least amount of effort, or "cost."

The Problem: The Lonely Neighbor

The paper points out a major flaw in how these delivery services used to work. Imagine two notes that are practically twins—maybe they both say, "I love playing soccer." They are sitting right next to each other in the pile.

Old-school OT methods looked at each note individually. If the cost to send "Note A" to the "Sports" warehouse was almost the same as sending it to the "Music" warehouse, the system would get confused. It might send "Note A" to Sports but "Note B" (its twin) to Music just because of a tiny, random difference.

The authors call this a lack of semantic consistency. It's like a teacher grading a test where two students who wrote the exact same answer get different grades just because the teacher was looking at them one by one instead of seeing they were a team. This confusion creates "noisy" labels, which messes up the whole sorting process.

The Solution: CAOT (The Neighborhood Watch)

The authors propose a new method called CAOT (Consistency-Aware Adaptive Optimal Transport). Instead of just looking at the distance between a note and a warehouse, CAOT adds a "neighborhood watch."

Here's how it works with a fun analogy:
Imagine you are trying to guess what flavor of ice cream a stranger likes.

Old Way: You ask the stranger, "Do you like chocolate?" They hesitate. You guess "Vanilla" because it's slightly closer to their answer.
CAOT Way: You look at the stranger's best friend standing right next to them. The friend is screaming, "CHOCOLATE!" CAOT realizes, "Hey, these two are inseparable! If the friend loves chocolate, the stranger probably does too."

CAOT does this by using a special attention mechanism. It builds a map of who is friends with whom based on their meaning. If two notes are semantically similar (they mean the same thing), CAOT forces them to get the same label. It combines the "global view" (where the note fits in the big picture) with the "local view" (who its neighbors are).

The Results: Sorting with Superpowers

The team tested this new method on eight different datasets, ranging from news headlines (AgNews) to technical questions (StackOverflow) and even tweets.

The Score: On the StackOverflow dataset, CAOT improved the accuracy by 5.01% compared to the previous best method. That's a huge jump in the world of sorting text!
The Consistency: In experiments, the old methods often assigned different labels to similar samples (the "twin" problem). CAOT fixed this, ensuring that neighbors got the same label.
The Speed: The paper notes that CAOT is also computationally efficient. While some older methods tried to solve the whole puzzle at once (which gets slow with huge data), CAOT works in smaller batches, making it faster and more scalable.

What It's NOT (And What It Rules Out)

It's important to know what this paper doesn't claim:

It's not magic for everything yet: The authors explicitly state that while the method works great on short texts, they suggest it could be generalized to long texts and images. They tested it on a few long-text datasets (like 20Newsgroups) and image datasets (like CIFAR-10), and it performed well, but the primary focus and "solved" status is for short text clustering.
It doesn't ignore the "Global" view: The paper argues against methods that only look at local neighbors or only look at the global structure. CAOT is designed to do both simultaneously.
It's not just "better guesses": The paper rules out simple "greedy" strategies where you just pick the closest label for each item individually. They show that without the global transport math, you get unreliable results.

How Sure Are They?

The authors are very confident in their numbers. They ran extensive experiments on real-world data.

They compared their method against 12 other top-tier methods (including things like TF-IDF, SimCSE, and RSTC).
They didn't just guess; they measured Accuracy (ACC) and Normalized Mutual Information (NMI).
They even did a "sensitivity analysis," changing the settings (hyperparameters) to make sure the method didn't break if things got slightly different. They found it was robust across balanced and imbalanced datasets (where some topics have way more notes than others).

The Bottom Line

The paper suggests that to sort short text effectively, you can't just look at the destination; you have to look at the company the text keeps. By teaching the sorting algorithm to respect the "friendships" between similar notes, CAOT creates much cleaner, more accurate groups than before. It's a step forward in making machines understand that context and consistency matter just as much as the words themselves.

Technical Summary: Harmonizing Global Structure and Local Consistency in Optimal Transport for Short Text Clustering

Problem Statement
Short text clustering is critical for organizing massive digital communications, yet existing methods face significant challenges in generating reliable pseudo-labels. Current approaches based on Optimal Transport (OT) improve upon greedy strategies by considering the global structure between sample and cluster distributions. However, these methods suffer from a critical limitation: they ignore sample-to-sample semantic consistency. Consequently, semantically similar samples may be assigned different pseudo-labels if their transport costs to candidate clusters are ambiguous or similar. This inconsistency introduces noisy pseudo-labels that misguide model optimization, particularly in feature spaces lacking clear category boundaries, leading to suboptimal clustering performance.

Methodology
The paper proposes CAOT (Consistency-Aware Adaptive Optimal Transport), a novel framework designed to harmonize global structural information with local semantic consistency. The framework operates within an Expectation-Maximization (EM) iterative loop comprising three core modules:

Semantic Similarity Construction Module (SSCM):
- This module captures instance-level semantic relationships. It first employs contrastive learning on augmented text views to enhance the discriminability of sample representations, ensuring a clear feature space.
- A novel instance-level attention network ( $G_h$ ) is then trained to generate an attention similarity matrix ( $S_{att}$ ). This network is optimized to assign high attention weights to samples belonging to the same cluster (identified via current pseudo-labels), thereby modeling the intrinsic semantic consistency between samples.
Pseudo-label Generation Module (PGM):
- This is the core innovation where CAOT is formulated. Unlike standard OT which minimizes transport cost solely based on sample-to-cluster distances, CAOT integrates the learned semantic similarity matrix ( $S = S_{cos} + S_{att}$ ) into the optimization objective.
- The optimization problem is defined as:
  $\min_{Q,b} \langle Q, -\log(P^{(0)}) \rangle + \varepsilon_1 H(Q) + \varepsilon_2 \Psi(b)^T \mathbf{1} - \varepsilon_3 \langle S, QQ^T \rangle$
- The term $\langle S, QQ^T \rangle$ acts as a regularizer that encourages the transport vectors of semantically similar samples to align, ensuring they receive identical pseudo-labels.
- The term $\Psi(b)$ replaces the hard uniformity constraint of standard OT with an adjustable penalty, allowing the model to adapt to varying degrees of class imbalance.
- The solution is obtained via an iterative method combining Taylor expansion and Lagrange multipliers.
Supervised Guidance Module (SGM):
- The reliable pseudo-labels generated by CAOT serve as supervisory signals to update the clustering network ( $G_p$ ) and the encoder ( $\Phi$ ) in the M-step, aligning predictive probabilities with the generated labels.

Key Contributions

Novel OT Formulation: The paper introduces CAOT, which is the first OT-based short text clustering method to explicitly incorporate sample-to-sample semantic consistency alongside sample-to-cluster global structure.
Instance-Level Attention Mechanism: A dedicated attention network is proposed to dynamically capture and model semantic relationships between samples, which are then integrated into the transport process.
Adaptive Imbalance Handling: The formulation includes a flexible penalty term for the target distribution, enabling the method to adapt to datasets with varying degrees of class imbalance without rigid constraints.
End-to-End Framework: The approach jointly optimizes representation learning, semantic consistency modeling, and clustering assignments in a unified EM framework.

Experimental Results
The authors conducted extensive experiments on eight benchmark datasets (including AgNews, StackOverflow, Biomedical, and various GoogleNews and Tweet subsets), covering balanced, imbalanced, and severely imbalanced scenarios.

Performance: The proposed method outperformed state-of-the-art baselines (including RSTC, SCPCL, FNSCC, and SCCL) on the majority of metrics (Accuracy and NMI). Notably, it achieved a 5.01% accuracy gain on the StackOverflow dataset compared to the best baseline.
Representation Quality: Visualizations and quantitative analysis of intra-cluster compactness and inter-cluster separability demonstrated that CAOT learns superior feature spaces compared to existing OT methods.
Ablation Studies: Removing the semantic consistency term or the contrastive learning component resulted in significant performance degradation, validating the necessity of both global structure and local consistency.
Efficiency: The method demonstrated computational advantages over existing OT methods (like RSTC) on large-scale datasets by operating on mini-batches rather than the full dataset, ensuring scalability.
Generalization: The framework was successfully extended to long-text datasets (e.g., 20Newsgroups) and image clustering tasks (STL-10, CIFAR), demonstrating cross-domain applicability.

Significance and Claims
The paper claims that the primary significance of this work lies in addressing the "blind spot" of existing OT methods regarding local semantic consistency. By correcting the misassignment of pseudo-labels for semantically similar neighbors, CAOT provides more reliable supervisory signals, leading to significantly improved clustering quality. The authors assert that the method is robust to hyperparameter variations and offers a general solution for clustering tasks beyond short texts, including long documents and images, without requiring manual hyperparameter tuning for class imbalance. The code is made available to facilitate reproducibility.