Zero-Shot Chinese Character Recognition via Global-Local Dual-Branch Alignment and Hierarchical Inference

Technical Summary: Zero-Shot Chinese Character Recognition via Global-Local Dual-Branch Alignment and Hierarchical Inference

Problem Statement
Zero-shot Chinese character recognition (ZS-CCR) addresses the challenge of identifying characters that were not present during training, a frequent occurrence in open-world scenarios such as ancient document digitization and variant character normalization. While Chinese characters are built from a limited set of radicals and spatial composition rules, existing retrieval-based methods often struggle with unseen characters. Current approaches, inspired by CLIP-style frameworks, typically encode character images and their Ideographic Description Sequences (IDS) into single global vectors for holistic matching. This paradigm often under-models local component differences and fine-grained compositional variations, leading to unstable discrimination in cases involving local substitutions or visually similar components. Furthermore, directly applying patch-token level fine-grained interaction across the entire candidate space introduces two critical issues: (1) structural operators in IDS (which describe spatial relations) lack concrete visual entities, introducing noise during local alignment; and (2) the computational cost of full-candidate fine-grained retrieval is prohibitively high for large-scale applications.

Methodology: Global-Local Hierarchical Perception Network (GL-HPN)
To address these limitations, the authors propose GL-HPN, a unified cross-modal alignment framework that jointly learns global and local representations through a dual-branch architecture.

1. Dual-Branch Representation:

   • Global Branch: Encodes character images and IDS sequences into holistic vectors. For images, a global average pooling is applied to the feature map; for IDS, token features are averaged. This branch captures stable sequence-level structural information to support efficient candidate recall.
   • Local Branch: Models fine-grained correspondence between image patches and IDS tokens. Image features are projected into a patch sequence, and IDS tokens are projected individually. This branch explicitly learns the alignment between specific visual regions and radical tokens.

2. Structure Filtering Mask (SFM):
   Recognizing that IDS contains both radical tokens (concrete visual components) and structural operators (abstract spatial relations), the authors introduce a Structure Filtering Mask. This mask explicitly suppresses structural operators during local similarity aggregation. By excluding non-entity tokens from the patch-token interaction, the model prevents scattered, non-informative responses from interfering with the discriminative contribution of radical tokens.

3. Training Objective:
   The model is trained using a multi-loss objective combining a Global Loss and a Local Loss.

   • The Global Loss uses a standard CLIP-style contrastive objective to align holistic image and text representations.
   • The Local Loss adopts a FILIP-style maximum-response aggregation, calculating similarity between image patches and valid (radical) tokens only, guided by the SFM.
   • A Curriculum-style weighting schedule is employed during training: the model initially emphasizes stable global alignment (high weight on Global Loss) and progressively strengthens local fine-grained discrimination (increasing weight on Local Loss) as training epochs advance.

4. Coarse-to-Fine Hierarchical Inference:
   To balance accuracy and efficiency, the inference process follows a two-stage strategy:

   • Coarse Recall: Global similarity scores are computed between the input image and the global representations of the entire candidate set. The Top-K candidates are selected.
   • Fine Reranking: Local fine-grained interaction is performed only on the Top-K candidates to compute reranking scores.
   • Fusion: Global and local posterior scores are normalized and fused via a parameter-free multiplicative operation to produce the final decision.

Key Contributions

• Global-Local Dual-Branch Framework: The paper proposes a novel architecture that balances holistic structural modeling with local component-level discrimination, avoiding the weaknesses of purely global or purely local approaches.
• Structure Filtering Mask (SFM): A mechanism to explicitly suppress noise caused by IDS structural operators in local fine-grained matching, thereby stabilizing local alignment.
• Coarse-to-Fine Hierarchical Inference: A strategy that restricts computationally expensive local interactions to a small set of high-confidence candidates, achieving a favorable trade-off between recognition accuracy and inference efficiency.

Experimental Results
The method was evaluated on handwritten Chinese character datasets (CASIA-HWDB1.0-1.1 and ICDAR2013) under various zero-shot splits (ranging from 500 to 2755 seen classes).

• Performance: GL-HPN achieved competitive Top-1 accuracy across all splits. Notably, it demonstrated superior performance in low-resource settings (e.g., 36.19% accuracy with only 500 seen classes), outperforming strong baselines like FT-CLIP, CCR-CLIP, and GRSTR by significant margins (e.g., +10.02% over FT-CLIP in the 500-class setting).
• Ablation Studies: Experiments confirmed that the dual-branch design alone is insufficient without the SFM (performance dropped without the mask). The SFM was shown to be critical for stabilizing local alignment and improving accuracy.
• Efficiency: The hierarchical inference strategy allowed the model to reach stable accuracy with a small candidate set (K=30 or K=50), significantly reducing inference latency compared to full-candidate local interaction, while maintaining high recall rates.

Significance
The paper claims that GL-HPN effectively addresses the specific challenges of zero-shot Chinese character recognition, particularly the noise introduced by abstract structural operators in IDS and the high cost of large-scale fine-grained retrieval. By decoupling global and local modeling and employing a hierarchical inference strategy, the method achieves robust generalization to unseen characters, especially in scenarios with limited training data, while substantially reducing the computational overhead associated with large-scale candidate retrieval.