💻 computer science

Multi-Head Attention based interaction-aware architecture for Bangla Handwritten Character Recognition: Introducing a Primary Dataset

This paper introduces a new balanced Bangla handwritten character dataset and proposes an interaction-aware hybrid deep learning architecture combining EfficientNetB3, Vision Transformer, and Conformer modules with multi-head cross-attention fusion, achieving state-of-the-art accuracy and strong generalization on both the new dataset and external benchmarks.

Original authors: Mirza Raquib, Asif Pervez Polok, Kedar Nath Biswas, Farida Siddiqi Prity, Saydul Akbar Murad, Nick Rahimi

Published 2026-04-15

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Mirza Raquib, Asif Pervez Polok, Kedar Nath Biswas, Farida Siddiqi Prity, Saydul Akbar Murad, Nick Rahimi

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a computer to read handwritten notes in Bangla (the language of Bangladesh and West Bengal). It's a bit like trying to teach a robot to recognize handwriting, but with a twist: Bangla handwriting is notoriously tricky.

Here is the simple breakdown of what this paper does, using some everyday analogies.

1. The Problem: The "Messy Handwriting" Nightmare

Think of Bangla characters like a set of very similar-looking twins. Some differ only by a tiny dot, a small curve, or a line placed slightly higher or lower (called a matra).

The Issue: Existing datasets (collections of practice images) were like a class photo taken of only 10-year-old boys. They didn't show enough variety. They missed the handwriting of elderly people, left-handed writers, or professionals.
The Result: When you trained a computer on these limited photos, it got confused. It couldn't tell the difference between a "K" and a "G" if the writer had a unique style. Also, the computer models used were like "one-trick ponies"—they were good at seeing small details but bad at understanding the whole picture, or vice versa.

2. The Solution Part A: The "Grand Library" (New Dataset)

The authors decided to build a massive, diverse library of handwriting samples to fix the training problem.

The Analogy: Instead of asking just one person to write a letter, they asked 1,700 different people to write.
The Mix: They included students from 6th grade all the way up to seniors, men and women, right-handed and left-handed writers, and people from different jobs.
The Collection: They gathered 50,700 images covering 78 different types of characters (basic letters, complex compound letters, and numbers).
Why it matters: Now, the computer isn't just learning from one style; it's learning from the entire spectrum of human handwriting. It's like teaching a child to recognize a dog by showing them a Chihuahua, a Great Dane, a Poodle, and a Golden Retriever, rather than just one Golden Retriever.

3. The Solution Part B: The "Super-Team" (The AI Model)

To recognize these tricky characters, the authors didn't just build one smart brain; they built a team of three experts who work together.

Imagine you are trying to identify a suspect in a lineup. You don't just ask one detective; you ask three:

The Micro-Observer (EfficientNet): This expert looks at the tiny details—the thickness of the ink, the curve of a stroke, and the exact shape of a letter. It's like a forensic artist looking at the fingerprint.
The Big Picture Thinker (Vision Transformer): This expert steps back and looks at the whole image. It understands how the parts of the letter relate to each other from a distance. It's like a detective who knows the general layout of the crime scene.
The Hybrid Detective (Conformer): This expert is a mix of both. It looks at the details and the big picture simultaneously.

The Secret Sauce: The "Round Table" (Cross-Attention)
In older models, these experts would just shout their opinions and the computer would pick the loudest one (or average them).
In this new model, the experts sit at a Round Table. They talk to each other!

The Micro-Observer says, "Hey, I see a tiny curve here."
The Big Picture Thinker says, "That curve fits perfectly with the structure I see over there."
The Hybrid Detective says, "Let's combine those thoughts."

This "conversation" (called Multi-Head Cross-Attention) allows them to refine their answers together. If one expert is unsure, the others help clarify. This is why the model is so good at telling apart those "twin" characters that only differ by a tiny dot.

4. The Results: A Masterpiece

When they tested this "Super-Team" on their new "Grand Library":

Accuracy: It got 98.84% correct. That's like getting almost every question right on a difficult exam.
Generalization: They also tested it on a different, external dataset (the CHBCR dataset), and it still scored 96.49%. This proves the model isn't just memorizing the answers; it actually learned how to read.
Visual Proof: They used a tool called Grad-CAM (like a heat map) to show where the computer was looking. The heat map lit up exactly on the important strokes of the letters, proving the computer was looking at the right things, not just guessing based on background noise.

Summary

This paper is about solving a difficult puzzle (reading messy Bangla handwriting) by doing two things:

Gathering a massive, diverse group of people to create a perfect practice set.
Building a team of AI experts who talk to each other to combine their strengths, rather than working in isolation.

The result is a system that is much more accurate and reliable, paving the way for better digital tools that can read and archive Bangla documents automatically.

1. Problem Statement

Handwritten Bangla Character Recognition (BHCR) faces significant challenges compared to printed text or other languages like English due to:

Structural Complexity: The Bangla script includes basic characters, compound characters (Juktobarno), and numerals. Many characters differ only by minor strokes, curves, or the placement of matras (vowel signs), leading to high inter-class similarity.
Data Limitations: Existing datasets often suffer from:
- Inequitable Class Distribution: Imbalanced sample sizes across classes.
- Limited Diversity: Data collected from narrow age groups or specific demographics, failing to capture real-world handwriting variations (e.g., left vs. right-handed, different writing styles).
- Lack of Documentation: Insufficient metadata regarding data collection methods and contributor demographics.
Architectural Gaps: Previous models often rely on single-stream CNNs (which miss long-range dependencies) or simple feature concatenation in hybrid models, failing to explicitly model the interaction between local stroke features and global structural relationships.

2. Methodology

The paper proposes a two-pronged solution: a new high-quality dataset and a novel hybrid deep learning architecture.

A. The New Dataset

The authors constructed a balanced, large-scale dataset to address data scarcity and diversity issues:

Scale: 50,700 handwritten samples across 78 classes (50 basic characters, 18 compound characters, 10 numerals).
Demographics: Collected from 1,700 contributors including students (grades 6–12), university students, and professionals.
Diversity: Includes a mix of genders (48% male, 52% female), age groups (10–35+), and writing preferences (88% right-handed, 12% left-handed).
Preprocessing: A structured pipeline was applied involving noise reduction (Gaussian filter), skew correction (Hough transform), morphological dilation, resizing to $128 \times 128$ , and normalization.

B. Proposed Architecture: MHCAF-Net

The core innovation is a Multi-Head Cross-Attention Fusion Network (MHCAF-Net) that operates in parallel branches to capture different feature types, fused via an interaction-aware mechanism.

Parallel Feature Extraction Branches:
- Local Feature Branch (EfficientNetB3 + CBAM): Uses EfficientNetB3 to extract fine-grained local stroke features. It is enhanced with a Convolutional Block Attention Module (CBAM) to refine channel and spatial attention, suppressing background noise and highlighting discriminative strokes.
- Global Feature Branch (Vision Transformer - ViT): Divides the image into patches and uses self-attention to capture long-range structural dependencies and global context.
- Hybrid Feature Branch (Conformer): Combines self-attention and convolution in a Macaron-style structure to model both local and global dependencies simultaneously.
Interaction-Aware Fusion (Multi-Head Cross-Attention):
- Instead of simple concatenation, the model employs a Multi-Head Cross-Attention mechanism.
- The feature vectors from the three branches ( $F_{eff}$ , $F_{vit}$ , $F_{conf}$ ) interact cyclically. For example, the EfficientNet features act as the Query while ViT features act as Key/Value, and vice versa.
- This allows the branches to refine each other's representations, explicitly modeling how local strokes relate to global structures.
Classification Head:
- The fused features pass through fully connected layers with Batch Normalization and Dropout, followed by a Softmax layer for 78-class classification.
- Trained using Categorical Cross-Entropy loss and the Adam optimizer.

3. Key Contributions

Dataset Creation: Introduction of a balanced, diverse, and large-scale Bangla handwritten character dataset (50,700 samples, 78 classes) with comprehensive demographic metadata.
Novel Architecture: Proposal of a parallel hybrid architecture integrating EfficientNetB3, ViT, and Conformer.
Interaction Mechanism: Development of a Multi-Head Cross-Attention Fusion module that enables heterogeneous feature streams to interact and refine one another, overcoming the limitations of simple feature stacking.
Comprehensive Evaluation: Extensive experiments including ablation studies, 5-fold cross-validation, and evaluation on external benchmarks.

4. Results

The proposed model demonstrated state-of-the-art performance across multiple metrics:

Internal Dataset Performance:
- Test Accuracy: 98.84%
- Precision/Recall/F1-Score: All exceeded 98.8%.
- Error Rate: Lowest at 1.16%.
- MCC & Kappa: Exceeded 98.8%, indicating strong agreement and balanced prediction.
External Benchmark (CHBCR Dataset):
- Achieved 96.49% accuracy on the external CHBCR dataset, demonstrating strong generalization capabilities despite domain shifts.
Ablation Studies:
- The full hybrid model with cross-attention outperformed single-branch models (e.g., standalone ViT or EfficientNet) and dual-branch concatenation models.
- The cross-attention fusion was proven critical, improving accuracy by ~0.5% over the best dual-branch baseline.
Stability: 5-fold cross-validation showed consistent performance (accuracy range 98.69%–98.91%), confirming robustness.
Explainability: Grad-CAM visualizations confirmed the model focuses on discriminative stroke regions and matras rather than background noise.

5. Significance

Advancing Low-Resource OCR: This work addresses the scarcity of high-quality, diverse datasets for Bangla, a low-resource language, facilitating better downstream applications like document digitization, archival preservation, and automated exam grading.
Architectural Insight: It demonstrates that interaction-aware fusion (cross-attention) is superior to simple concatenation for fine-grained classification tasks where local details and global structure are tightly coupled.
Real-World Applicability: By training on a dataset that reflects real-world demographic and stylistic variations (age, handedness, education level), the model is more robust to the "natural" variability of human handwriting, making it suitable for practical deployment in diverse environments.