Restoration-Guided Kuzushiji Character Recognition Framework under Seal Interference

This paper proposes a three-stage restoration-guided framework (RG-KCR) that effectively mitigates seal interference in pre-modern Japanese documents by integrating character detection, image restoration, and classification, thereby significantly improving recognition accuracy compared to existing methods.

The Gist

Imagine you've found an ancient, dusty diary from 300 years ago. The handwriting is beautiful but messy—like a fast, flowing cursive script that nobody uses anymore. This is Kuzushiji, the cursive writing style of pre-modern Japan.

Now, imagine someone tried to read this diary, but every few pages, there's a giant, red stamp (a seal) slapped right over the words. In the old days, these stamps were like signatures or official seals of ownership, but today, they are the biggest headache for computers trying to read the text. They cover the letters, making them look like a messy red blob.

This paper introduces a new "super-reader" system called RG-KCR (Restoration-Guided Kuzushiji Character Recognition) that solves this specific problem. Here is how it works, broken down into three simple steps:

Step 1: The "Spotter" (Finding the Words)

First, the system needs to know where the words are, even if they are covered by red stamps.

• The Analogy: Think of this like a game of "Where's Waldo?" but instead of finding Waldo, the computer is looking for individual Japanese characters.
• What they did: They trained a very sharp-eyed AI (called YOLOv12) to hunt down every single character. Even if a red stamp is sitting on top of a character, this AI is smart enough to say, "Hey, there's a letter hiding under that red ink!" and draw a box around it.

Step 2: The "Magic Eraser" (Cleaning the Image)

Once the system knows where the characters are, it needs to clean them up before trying to read them.

• The Analogy: Imagine you spilled red paint on a page of text. You wouldn't try to read the letters while the paint is still there. You'd use a magic eraser to gently lift the red paint off without smudging the black ink underneath.
• What they did: They created a special, fast algorithm that acts like a digital magic eraser. It looks for pixels that are "too red" (the seals) and gently paints over them with the surrounding paper texture. It's like a painter filling in a hole in a wall so the picture looks whole again. Crucially, this step doesn't require the computer to "learn" anything new; it just uses math to spot the red and fix it instantly.

Step 3: The "Translator" (Reading the Clean Text)

Now that the red stamps are gone and the characters are clean, the system can finally read them.

• The Analogy: This is like handing a clean, clear page to a master translator who knows thousands of ancient dialects.
• What they did: They fed the cleaned-up character images into a powerful AI called Metom. Because the red stamps were removed in Step 2, Metom can now read the characters with much higher accuracy. It translates the ancient, messy cursive into modern, readable Japanese.

Why is this a big deal?

Before this paper, existing systems were like trying to read a book through a foggy, red-tinted window. They would often guess wrong or get confused when seals were present.

The authors tested their new three-step process and found:

1. The Spotter found 98% of the characters correctly, even with seals on them.
2. The Magic Eraser successfully removed the red stamps, making the text look almost like new.
3. The Translator got much smarter after the cleaning step. Its accuracy jumped from about 93% to 95%.

The Final Result

The system doesn't just output a list of words; it takes the original ancient image, removes the red stamps, and overlays the modern Japanese translation right on top of the original text. It's like having a pair of glasses that instantly cleans up the red ink and whispers the meaning of the ancient words directly into your ear.

In short: They built a pipeline that finds the messy text, erases the annoying red stamps, and then reads the clean text, making history much easier for modern people to understand.

Technical Summary

1. Problem Statement

Kuzushiji is a highly cursive, pre-modern Japanese writing style used in historical documents. While modern AI systems have achieved reasonable success in recognizing Kuzushiji from clean documents, they face significant challenges when seals (hanko) overlap with characters.

• The Challenge: Seals are ubiquitous in pre-modern Japanese documents, serving as signatures or ownership marks. They are typically stamped in red ink and often occlude or overlap with the text.
• Current Limitations: Existing state-of-the-art (SOTA) systems (e.g., Fuminoha, NDLkotenOCR-Lite, Metom) struggle to maintain accuracy when seals interfere with characters. The red ink creates visual noise and occlusion that confuses standard OCR and classification models, leading to a sharp drop in recognition performance.
• Gap: There is a lack of specialized frameworks that explicitly address seal interference through a combination of detection, restoration, and classification.

2. Methodology: The RG-KCR Framework

The authors propose a Restoration-Guided Kuzushiji Character Recognition (RG-KCR) framework consisting of three distinct stages:

Stage 1: Kuzushiji Character Detection

• Goal: Locate individual character instances within the document.
• Approach: The authors evaluated various object detection models (RT-DETR, YOLOv9, YOLOv10, YOLO11, YOLOv12).
• Selection: YOLOv12-medium was selected as the optimal detector.
  • It achieves high precision (98.0%) and recall (93.3%) even on images with seal interference.
  • Unlike line-level detectors (which often produce duplicate boxes on complex layouts), this stage uses character-level detection to generate precise bounding boxes for each glyph.
• Note: Restoration is not applied before this stage because the detector is robust enough to handle seals without preprocessing.

Stage 2: Kuzushiji Document Restoration (Seal Removal)

• Goal: Remove red seal artifacts from the regions identified in Stage 1 to improve classification accuracy.
• Approach: A training-free, color-based seal removal algorithm.
  • Detection: Since seals are red, the algorithm identifies seal pixels based on channel intensity thresholds: $R \ge \tau_r$, $R \ge \tau_{rg} \cdot G$, and $R \ge \tau_{rb} \cdot B$.
  • Refinement: Morphological dilation is applied to the binary mask to account for ink bleeding.
  • Inpainting: The detected seal regions are reconstructed using image inpainting (Telea's fast marching method or Navier-Stokes approach) to propagate surrounding textures and structural cues into the masked areas.
• Efficiency: This stage is computationally efficient and does not require training a neural network, making it suitable for resource-constrained environments.

Stage 3: Kuzushiji Character Classification

• Goal: Recognize the specific Kuzushiji character within the restored bounding box.
• Approach: The cropped character images from the restored document are fed into Metom, a Vision Transformer (ViT)-based classifier capable of handling over one million character classes.
• Output: The predicted Unicode code point is mapped to a modern Japanese character and overlaid onto the original document layout for human-readable interpretation.

3. Key Contributions

1. Novel Framework: Introduction of the RG-KCR framework, the first system specifically designed to mitigate seal interference in Kuzushiji recognition via a restoration-guided pipeline.
2. Efficient Restoration Algorithm: Development of a training-free, color-based seal removal method that effectively reduces seal artifacts without the high inference cost of deep learning-based restoration models.
3. Dataset Construction:
   • Created a Kuzushiji Character Detection Dataset (1,000 images from 13 books) with manually corrected annotations to ensure reliability.
   • Created a Synthetic Seal Dataset by overlaying real seal images onto documents to simulate interference patterns for rigorous evaluation.
   • Constructed a Classification Test Set containing 17,982 character instances with synthetically overlaid seals.
4. Comprehensive Evaluation: Conducted ablation studies and quantitative evaluations demonstrating the direct impact of document restoration on classification accuracy.

4. Experimental Results

The framework was evaluated on the constructed datasets using standard metrics (Precision, Recall, PSNR, SSIM, Top-1 Accuracy).

• Detection (Stage 1):
  • YOLOv12-medium achieved 98.0% Precision and 93.3% Recall on the test set, outperforming other YOLO variants and DETR-based models.
• Restoration (Stage 2):
  • Optimal hyperparameters were found to be $\tau_r = 90$ and $(\tau_{rg}, \tau_{rb}) = (1.3, 1.3)$.
  • Achieved a PSNR of ~34.13 dB and SSIM of ~0.9750 on the test set, indicating high-fidelity reconstruction of the text regions.
• Classification (Stage 3):
  • Ablation Study: When using Metom directly on raw (seal-interfered) images, the Top-1 accuracy was 93.45%.
  • With the proposed Stage 2 restoration, the Top-1 accuracy improved to 95.33% (+1.88% gain).
  • Top-5 accuracy improved from 97.46% to 98.62%.
  • The restoration stage added only 0.51 seconds per image, a negligible cost for the accuracy gain.

5. Significance and Future Work

• Significance: This work addresses a critical bottleneck in digital humanities: the inability of current OCR systems to handle the ubiquitous red seals in historical Japanese documents. By decoupling detection, restoration, and classification, the framework offers a modular and efficient solution that significantly boosts recognition accuracy without requiring massive retraining of classification models.
• Limitations & Future Directions:
  • The current system outputs recognized characters overlaid on the image but does not yet generate continuous, ordered text.
  • Future work aims to incorporate character ordering and layout analysis to reconstruct the full reading order (which can be complex in Kuzushiji) and enable end-to-end text transcription.
  • The framework is open-sourced, facilitating further research and application in historical document preservation.