🤖 AI

Logographic Character Visual Pretraining via Semantic-based Contrastive Learning

This paper proposes a novel semantic-based contrastive learning pre-training strategy that leverages multi-modal visual and contextual semantics to enhance deep visual representations for logographic character recognition, effectively addressing performance limitations caused by imbalanced and rare character datasets.

Original authors: Daqian Shi, Wei Cao, Xiaoyu Zheng, Lida Shi, Xiaolei Diao, Cedric M John

Published 2026-08-04

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Daqian Shi, Wei Cao, Xiaoyu Zheng, Lida Shi, Xiaolei Diao, Cedric M John

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to recognize every single character in the world, from the letters in your name to the ancient symbols carved on stone tablets. This is the world of computer vision, a branch of artificial intelligence where machines learn to "see" and understand images. For a long time, these robots have been great at spotting simple things like cats or cars, but they struggle with logographic characters—writing systems where one symbol represents a whole word or idea, like Chinese Hanzi or Japanese Kanji. The problem isn't just that there are thousands of these characters; it's that some are used every day (like "the" or "water"), while others are so rare they might appear only once in a thousand years. It's like trying to learn a language where you have a million copies of the word "apple" but only one copy of "zebra." The robot gets confused, over-focusing on the common words and forgetting the rare ones. To fix this, scientists use a technique called contrastive learning, which is like a game of "spot the difference." The robot is shown two pictures and asked, "Are these the same?" If they are, it pulls them closer together in its brain; if not, it pushes them apart. But there's a catch: the robot usually treats every "different" pair as equally different, which doesn't work well when some differences are subtle and others are obvious.

This paper introduces a clever new way to teach these robots how to understand rare and common characters better by giving them a reference: context. The authors, a team of researchers from universities in the UK and China, realized that just looking at the shape of a character isn't enough. In human language, we understand words not just by how they look, but by the company they keep. For example, the word "orchard" makes you think of trees and fruit, while "garden" makes you think of flowers and paths. Even though they are different words, they are semantically close. The researchers noticed that logographic characters work the same way: a character that looks like a fish often means "fish," and characters with similar meanings often share visual parts. They proposed a method called Semantic-based Contrastive Learning. Instead of just telling the robot, "These two are different, push them apart," they use a language model (a super-smart text predictor) to tell the robot how different they are. If two characters are semantically close (like "orchard" and "garden"), the robot is told to keep them somewhat close, even if they look different. If they are totally unrelated, it pushes them far away. This creates a "soft" map of relationships rather than a rigid black-and-white list.

The team tested this idea on several datasets, including handwritten Chinese characters, Japanese historical texts, and real-world street signs. They found that their new method significantly outperformed existing state-of-the-art techniques, especially when the data was messy and unbalanced. For instance, on a dataset called HWDB1.1 with a severe imbalance (where the most common class had 100 times more samples than the rarest), their method achieved an accuracy of 83.46% using a standard ResNet18 backbone, beating the previous best contrastive learning method (SimCLR) which only reached 56.68%. Even on the more balanced K-Kanji dataset, their approach hit 95.30%, surpassing the next best method by a clear margin. The researchers also ran "ablation studies" (experiments where they turned off parts of their system) to prove that both the visual learning and the contextual reference were necessary; using only one or the other resulted in lower scores. They also discovered that simply trying to match the exact distance between words (a method called MSE) didn't work as well as matching the pattern of relationships (distribution-level alignment). By treating the relationship between characters as a flexible, graded spectrum rather than a fixed rule, the robot learns to recognize even the rarest, most obscure characters with much greater confidence. This suggests that by borrowing the way humans understand context, we can build smarter, more balanced AI for reading the world's most complex writing systems.

Technical Summary: Logographic Character Visual Pretraining via Semantic-based Contrastive Learning

Problem Statement

Deep learning-based character vision tasks, such as text recognition and image denoising, typically achieve optimal performance only when trained on large, balanced datasets. However, real-world datasets for logographic languages (e.g., Chinese Hanzi, Japanese Kanji, ancient Egyptian hieroglyphs) suffer from severe data imbalance. This imbalance arises from significant differences in character usage frequency and the continuous emergence of new or rare characters through archaeological discoveries.

Existing recognition methods, often rooted in image classification, struggle with these long-tailed distributions. While contrastive learning (e.g., SimCLR) has been applied to mitigate category imbalance, current approaches rely on binary positive/negative pair formulations. These methods treat all negative samples as equally dissimilar (fixed distance), failing to distinguish between "hard" negatives (visually or semantically similar but distinct characters) and "trivial" negatives. Consequently, learning efficiency is reduced, and representations for low-frequency characters remain under-optimized. Furthermore, existing methods primarily focus on pixel-level features, neglecting the rich contextual semantics inherent in logographic systems.

Methodology

The authors propose a novel pre-training strategy that integrates visual semantics and contextual semantics to enhance deep visual representations, specifically for imbalanced logographic character datasets. The core innovation is a semantic-based contrastive learning framework that replaces binary labels with soft semantic supervision derived from language models.

1. Dual-Semantic Representation

The framework models two distinct semantic spaces:

Visual Semantics ( $V_s$ ): Extracted from character images using a vision encoder (e.g., ResNet). The encoder processes augmented character images to produce embedding vectors representing visual features.
Contextual Semantics ( $C_s$ ): Extracted from character labels using a pre-trained language model (e.g., BERT). Instead of treating text embeddings as absolute targets, the model leverages the language model's capability to capture relative semantic relationships between characters.

2. Distribution-Level Contrastive Objective

Rather than regressing pairwise similarity scores directly (which can be sensitive to scale and non-isometric distributions), the method aligns the semantic neighborhood distributions of the two modalities.

Soft Label Generation: For each anchor character, the contextual semantic similarities to other characters in a mini-batch are converted into a conditional probability distribution ( $P_C$ ) using a temperature-scaled softmax. This distribution represents the relative semantic relevance of neighbors.
Visual Distribution Alignment: Similarly, a visual semantic distribution ( $P_V$ ) is computed from the visual embeddings.
Loss Function: The primary objective is to minimize the cross-entropy (equivalent to Kullback–Leibler divergence) between the contextual distribution ( $P_C$ ) and the visual distribution ( $P_V$ ). This forces the visual embedding space to preserve the relative semantic ordering defined by the language model, effectively using contextual semantics as soft labels.

3. Difficulty-Aware Focal Reweighting

To address the issue where easy, trivial pairs dominate the loss in long-tailed datasets, the authors introduce a focal reweighting strategy inspired by focal loss.

The loss is reweighted based on the current prediction confidence of the visual model.
Pairs that are already well-aligned (easy) are down-weighted, while misaligned, informative "hard" pairs are emphasized.
This adaptive mechanism ensures that the model focuses learning capacity on difficult semantic relationships, which are crucial for distinguishing rare or similar-looking characters.

4. Joint Optimization

The pre-training phase jointly optimizes the contextual semantic loss (distribution alignment) and a standard classification loss (cross-entropy) to anchor the learned representations to the specific recognition task.

Key Contributions

Novel Pre-training Strategy: The paper introduces a method to enhance deep representations for imbalanced character categories by integrating semantic-based contrastive learning. It moves beyond binary contrastive learning to utilize continuous semantic relevance.
Distribution-Level Alignment Framework: A framework is proposed that aligns visual and contextual semantics at the distribution level. By modeling relative semantic relationships rather than absolute distances, it enables more effective representation learning for logographic characters.
Empirical Validation: Extensive experiments across four diverse datasets (HWDB1.1, ICDAR2013, CTW, and K-Kanji) with varying degrees of imbalance demonstrate the method's superiority over state-of-the-art pre-training and long-tail classification techniques.

Experimental Results

The proposed method was evaluated on four datasets covering handwritten Chinese, scene text, and historical Japanese Kanji, under various imbalance factors (IF = 10, 50, 100).

Comparison with Pre-training Methods: The proposed method (Ours $_{FR}$ ) consistently outperformed established contrastive learning baselines (SimCLR, MoCo, BYOL, SwAV, SimSiam) and multimodal models (VL-BERT, CLIP). For instance, on the highly imbalanced HWDB-LT100 dataset with a ResNet18 backbone, the proposed method achieved 83.46% accuracy, significantly surpassing SimCLR (56.68%) and MoCo (58.96%).
Comparison with Long-Tail Methods: The approach also outperformed specialized long-tail image classification methods (CMO, OTmix, NCL++, ProCo), indicating that integrating contextual semantics provides superior robustness compared to methods relying solely on visual sample distributions.
Downstream Task Improvement: When applied as a pre-training step for various SOTA OCR models (including CNNs like VGG16 and Transformers like ViT), the method yielded consistent accuracy improvements across all tested architectures. For example, applying the pre-training to DenseNet on the CTW dataset resulted in a 2.07% accuracy gain.
Ablation Studies:
- Loss Components: Using only contextual loss significantly outperformed using only classification loss, and the combination of both yielded the best results.
- Alignment Strategy: The proposed distribution-level alignment (KL-divergence) significantly outperformed a point-wise MSE regression approach, confirming that learning relational structures is more effective than regressing absolute similarity values in non-isometric, long-tailed spaces.

Significance

The paper claims that its primary significance lies in addressing the critical bottleneck of data imbalance in logographic character recognition. By leveraging the consistency between visual structures and contextual meanings, the proposed method provides a mechanism to learn robust representations even when training samples for specific characters are scarce. The authors assert that this approach offers a more scalable and robust solution for real-world applications involving historical texts, scene characters, and low-resource languages, where traditional balanced training is unfeasible. The work demonstrates that incorporating external semantic knowledge (via language models) as soft supervision can effectively guide visual representation learning, reducing reliance on large, balanced visual datasets.