💻 computer science

STR-DC: Spatio-Temporal Reliability Learning for Robust Deep Clustering of Japanese Handwritten Characters

The paper proposes STR-DC, a deep clustering framework that enhances robustness in Japanese handwritten character recognition by integrating calibrated semantic confidence with cross-epoch temporal feature stability to filter out unreliable pseudo-labels, achieving state-of-the-art performance on multiple benchmarks.

Original authors: Yang Liu, Ziyang Li

Published 2026-10-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Yang Liu, Ziyang Li

Original paper licensed under CC BY 4.0 (https://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a library where millions of handwritten notes have been gathered, but no one knows what any of them say. The handwriting varies wildly from person to person; some writers are neat, others messy, and some strokes are so faint they are barely visible. To make this collection useful, someone must sort these notes into groups based on the character they represent. Doing this by hand is a monumental task, requiring experts to look at every single image and decide which group it belongs to. In the world of computer science, researchers have developed methods to let machines do this sorting automatically, a process known as clustering. These machines learn to recognize patterns and group similar images together without being told the correct answers in advance. However, this is a tricky business. When a machine guesses the group for a difficult or blurry image, it might be very confident in its wrong answer. If the machine then uses that wrong guess to teach itself, it reinforces its own mistake, creating a cycle of error that makes the final groups inaccurate.

The challenge becomes even more complex with Japanese handwritten characters. Many of these characters share similar shapes or local stroke patterns, and a single character can look completely different depending on who wrote it. A faint line or a missing stroke can make two different characters look identical to a computer. Recent approaches tried to solve this by teaching the machine to be more careful about how much it trusts its own guesses, a concept called confidence calibration. This helps the machine realize when it is unsure, but it is not a perfect solution. A machine can still be confidently wrong about a character that looks like another, or it might settle on a wrong group and stay there because the image is stable enough to look consistent, even if the meaning is wrong.

To address these lingering problems, researchers Yang Liu and Ziyang Li have developed a new system called STR-DC. Instead of relying on a single moment of judgment, this system looks at the history of the machine's learning process. The researchers realized that a reliable answer should not only look correct right now but should also remain consistent as the machine learns over time. They built a framework that checks two things for every image it tries to sort. First, it asks: does the machine have strong evidence that this image belongs to a specific character group? This is the semantic check. Second, it asks: has the machine's understanding of this image remained steady as it trained? This is the temporal check. By requiring an image to pass both tests, the system filters out the tricky cases that might fool a simpler method.

The researchers tested this approach on three large collections of Japanese handwritten characters, known as K49, ETL8-Hira75, and ETL9-Hira71. These datasets contain thousands of images with varying levels of difficulty, including incomplete strokes and different writing styles. When they ran their new system against existing methods, the results were clear. On the K49 dataset, which contains 49 different character classes, their method achieved an accuracy of 57.68 percent. This was a significant improvement over the previous best method, which scored 52.16 percent. The improvement was even more striking on the ETL8-Hira75 dataset, where their system reached 78.46 percent accuracy compared to 68.96 percent for the next best approach. On the third dataset, ETL9-Hira71, they achieved 82.11 percent accuracy. These numbers indicate that the new system is better at grouping the handwritten forms correctly, even when the visual differences are subtle.

To understand why this worked, the researchers looked closely at what the system was rejecting. They found that images with low stability—those where the machine's internal representation changed significantly from one training step to the next—were often the ones that were being sorted incorrectly. These unstable images tended to be the blurry, faint, or ambiguous ones that caused confusion. By removing these unstable candidates and only using the steady, confident ones to teach the machine, the system avoided the trap of learning from its own mistakes. The researchers also checked if this idea worked on general images, not just handwriting. They tested the system on standard image sets containing everyday objects and animals, and it performed better than other leading methods there as well, suggesting that the logic of checking for both confidence and stability is useful for many types of visual data.

The study suggests that for machines to learn effectively from unlabeled data, they need a way to verify their own reliability over time, not just in a single snapshot. The new method does not claim to solve every problem in handwriting recognition, and the researchers note that it still requires some manual tuning and adds a bit of extra work to the training process. However, by combining a check for how sure the machine is with a check for how steady its understanding is, they have created a more robust way to organize these difficult collections. This approach offers a clearer path forward for sorting vast archives of handwritten documents, turning a chaotic pile of images into an organized resource that can be studied and understood.

Technical Summary: STR-DC for Robust Deep Clustering of Japanese Handwritten Characters

Problem Statement
The paper addresses the challenge of unsupervised deep clustering for Japanese handwritten characters. While clustering can organize unlabeled collections for subsequent annotation and retrieval, Japanese handwriting presents specific difficulties: visually similar categories, diverse writing styles, incomplete strokes, and degraded image quality. In standard deep clustering pipelines, early confusion between similar character shapes often leads to incorrect pseudo-labels. These errors are reinforced during self-training, resulting in confirmation bias and incorrect cluster boundaries. Existing methods, such as Calibrated Deep Clustering (CDC), mitigate overconfidence by introducing confidence calibration; however, they rely solely on the current prediction snapshot. Consequently, samples with incomplete strokes or shared local structures may still receive deceptively high probabilities, leading to confident but incorrect assignments. Conversely, methods relying solely on temporal stability (e.g., LFSS) may retain samples that are stable across epochs but semantically weak or incorrectly assigned to a consistent wrong cluster.

Methodology: STR-DC Framework
The authors propose STR-DC (Spatio-Temporal Reliability-Guided Deep Clustering), a framework designed to select reliable pseudo-labels by combining calibrated semantic confidence (spatial reliability) with cross-epoch feature stability (temporal reliability). The framework operates through the following mechanisms:

Architecture: STR-DC utilizes a dual-head architecture consisting of a clustering head and a calibration head, built upon a shared feature extractor. It employs weak ( $W$ ) and strong ( $A$ ) data augmentations. The weak view is used for reliability estimation and pseudo-label generation, while the strong view is used for optimizing the clustering head.
Spatial Semantic Reliability (Calibrated Confidence):
- The calibration head estimates more reliable confidence scores for the weakly augmented view.
- A class-wise calibrated selection strategy is applied: for each predicted cluster, samples are ranked by their calibrated probabilities, and a dynamic number of top samples are selected to form a set of semantically confident candidates ( $S_{sem}$ ).
- This step ensures that selected samples have strong evidence for their predicted character cluster within the current prediction space.
Temporal Reliability (Feature Stability):
- To verify the consistency of these candidates, STR-DC measures the stability of feature representations across training epochs.
- For each sample, the cosine similarity is computed between its current weak-view feature representation ( $z_i^{w,(t)}$ ) and its predecessor feature from the previous epoch ( $z_i^{w,(t-1)}$ ).
- An adaptive, percentile-based threshold is used to filter out samples with low temporal stability (high feature drift), generating a stability mask ( $m_i^{stab}$ ).
Spatio-Temporal Reliability Fusion:
- The core innovation is a strict conjunctive fusion rule: a sample is retained for self-training only if it satisfies both the calibrated semantic confidence criterion and the temporal stability criterion ( $m_i^{rel} = m_i^{sem} \land m_i^{stab}$ ).
- This intersection ensures that the model learns from samples that are not only semantically plausible but also dynamically stable, effectively filtering out high-confidence false positives and stable but semantically weak samples.
Optimization: The clustering head is optimized using a reliable clustering loss ( $L_{rel}$ ) computed only on the final reliable set ( $S_{rel}$ ), while the calibration head is trained using a calibration objective ( $L_{cali}$ ) to maintain accurate confidence estimation.

Key Contributions

Novel Framework: Introduction of STR-DC, which explicitly addresses the limitations of using only calibrated confidence or only feature stability for pseudo-label selection in Japanese handwritten character clustering.
Complementary Reliability Signals: The paper demonstrates that calibrated semantic confidence and cross-epoch feature stability provide complementary evidence. The former identifies category-level evidence, while the latter verifies representation consistency.
Strict Fusion Mechanism: The proposal of a strict "AND" fusion rule that retains only samples satisfying both criteria, thereby reducing the influence of ambiguous character assignments during self-training.
Generalization: The method is shown to generalize beyond Japanese characters to general image clustering benchmarks, suggesting the reliability mechanism is robust across different visual domains.

Experimental Results
The authors evaluated STR-DC on three Japanese handwritten character datasets (K49, ETL8-Hira75, ETL9-Hira71) and four general image benchmarks (CIFAR-10, CIFAR-20, ImageNet-10, ImageNet-Dogs).

Japanese Handwritten Characters: STR-DC achieved state-of-the-art performance across all metrics (ACC, NMI, ARI).
- On K49, it achieved 57.68% ACC, outperforming the calibrated-confidence baseline (CDC) by 5.52%.
- On ETL8-Hira75, it achieved 78.46% ACC, an improvement of 9.50% over CDC.
- On ETL9-Hira71, it achieved 82.11% ACC, exceeding CDC by 1.83%.
General Image Benchmarks: STR-DC consistently outperformed state-of-the-art methods (including ProPos, CoNR, DMICC, BYOL, MoCo-v2, and LFSS) on CIFAR and ImageNet subsets, with particularly notable gains on fine-grained datasets like ImageNet-10 (97.2% ACC).
Ablation Studies: Results confirmed that neither component alone is sufficient. The "Calibrated Confidence Only" variant performed well but was outperformed by the full model, while the "Feature Stability Only" variant performed significantly worse, particularly on ETL8-Hira75 (31.00% ACC vs. 78.46% for the full model), validating the necessity of the semantic prior.
Visualization: t-SNE and distribution analyses showed that correctly clustered samples generally exhibit higher cross-epoch feature stability, while low-stability samples often appear near cluster boundaries or in sparse regions.

Significance and Claims
The paper claims that STR-DC provides a more robust approach to unsupervised clustering for Japanese handwritten characters by mitigating the risks of confirmation bias inherent in self-training. By requiring both semantic evidence and temporal consistency, the method organizes unlabeled character collections more effectively, facilitating subsequent inspection and annotation. The authors modestly note that while the method is effective, it relies on short-term feature stability between adjacent epochs and requires empirical selection of stability thresholds. They suggest future work could explore more adaptive temporal modeling and efficient memory updates for large-scale datasets. The work is positioned as a step toward organizing unlabeled Japanese handwritten character collections, rather than a complete solution for all handwriting recognition tasks.

Technical Summary: STR-DC for Robust Deep Clustering of Japanese Handwritten Characters

More like this