Barnamala: Parameter-Efficient Handwritten Devanagari Recognition at Benchmark Saturation

Technical Summary: Barnamala – Parameter-Efficient Handwritten Devanagari Recognition at Benchmark Saturation

Problem Statement
The DHCD dataset serves as the standard benchmark for 46-class handwritten Devanagari character recognition. Recent state-of-the-art (SOTA) models, such as MallaNet (17.32 M parameters) and Mishra et al.'s architecture, have achieved accuracies clustering tightly around 99.71–99.72%. At this level, incremental gains are measured in hundredths of a percent, yet existing models remain significantly larger than what is feasible for on-device deployment. Furthermore, prior evaluations often rely on raw accuracy rankings without rigorous statistical testing (e.g., exact McNemar tests), failing to account for the fact that near-perfect accuracy on a 13,800-sample test set renders raw rankings statistically insignificant. The core problem addressed is whether a compact model can match SOTA performance and whether the benchmark itself has reached an intrinsic saturation point where further architectural scaling yields no meaningful improvement.

Methodology
The authors propose Barnamala, a compact convolutional neural network designed for single-channel 32×32 grayscale images.

• Architecture: Barnamala is a pre-activation Squeeze-and-Excitation (SE) Residual Network. It consists of a stem convolution followed by three stages of PreAct SE-ResBlocks with channel widths of 40, 80, and 160, and depths of 2, 2, and 2 respectively. Stages 2 and 3 utilize stride-2 downsampling. The total parameter count is 1.11 M.
• Training Strategy:
  • Data Augmentation: Recognizing that Devanagari is not horizontally symmetric, horizontal flipping is strictly excluded. Instead, the pipeline uses affine perturbations (rotation, translation, scale, shear), elastic deformations, random erasing, and a mix of Mixup and CutMix (activated with 50% probability per batch).
  • Knowledge Distillation: The authors investigate whether a student model benefits from a teacher ensemble. They utilize an ensemble of 15 teachers (varying in architecture width/depth and augmentation intensity) trained without distillation. The student is trained using a combination of hard cross-entropy (with label smoothing) and a temperature-scaled KL divergence against the mean softmax of the teacher ensemble.
  • Optimization: AdamW optimizer with a cosine learning rate schedule, 5-epoch linear warm-up, and an Exponential Moving Average (EMA) of weights for the final checkpoint.
• Evaluation Protocol:
  • Statistical Rigor: The authors employ the exact two-sided McNemar test on per-sample binary correctness indicators to compare models, accompanied by Wilson score intervals for confidence bounds. This addresses the issue of "noise floor" accuracy where raw percentages are indistinguishable.
  • Robustness & Transfer: Evaluation includes zero-shot and fine-tuned performance on the CMATERdb digit dataset and zero-shot performance on the NHCD and Prashanth datasets. Corruption robustness is tested against Gaussian noise, blur, and contrast reductions (ImageNet-C protocol).

Key Contributions

1. Compact SOTA Performance: Barnamala achieves 99.73% accuracy on the 46-class DHCD test set, matching the highest reported results while being 15.6× smaller (1.11 M vs. 17.32 M parameters) and 9.5× faster on CPU than the prior SOTA (MallaNet).
2. Statistical Parity Without Distillation: McNemar testing reveals that Barnamala is statistically indistinguishable from the 17.32 M parameter baseline (p = 0.345). Crucially, this parity holds even for the supervised-only version of Barnamala (trained without teacher signals), indicating that knowledge distillation is not the primary driver of the performance gain in this saturated regime.
3. Benchmark Saturation: The study identifies an intrinsic error floor of 11 images. Every configuration tested, including large teacher ensembles and various student seeds, fails on these exact same 11 samples. This suggests the benchmark is saturated, and further progress requires a harder evaluation set or different modeling assumptions rather than parameter scaling.
4. Ablation of Distillation Factors: The authors demonstrate that neither the cleanliness of the distillation target (clean vs. TTA-softened) nor the ensemble size (9 vs. 15 teachers) yields a statistically significant improvement once noise is properly accounted for.
5. Superior Robustness and Transfer: Despite its small size, Barnamala exhibits significantly better robustness to corruptions (mean corruption accuracy 75.7% vs. 38.7% for the baseline) and transfers more effectively to out-of-distribution datasets (e.g., NHCD and CMATERdb) compared to larger baselines.

Results

• Accuracy: Barnamala (mean of 5 seeds) achieves 99.735% accuracy with 36.6±2.1 errors. The baseline MallaNet (reproduced) has 40 errors (99.710%). The difference is not statistically significant (McNemar p = 0.345).
• Efficiency: Barnamala requires 164M MACs compared to 2434M for MallaNet. Inference latency on a single-threaded CPU is 8.28 ms vs. 78.8 ms.
• Distillation Impact: While distillation slightly reduced the error count (from 39.8 to 36.6), the difference was not statistically significant, and the "clean" vs. "TTA" target methods produced statistically indistinguishable results.
• Transfer Learning:
  • CMATERdb (Digits): Zero-shot accuracy of 76.6% (vs. 62.7% for a supervised-only baseline); fine-tuned accuracy of 97.8%.
  • NHCD (46 classes): Zero-shot accuracy of 78.92% (vs. 23.99% for MallaNet), highlighting a massive gap in generalization capability despite identical in-distribution performance.
• Corruption Robustness: Barnamala outperforms the baseline significantly in blur (97.0% vs. 37.6%) and contrast (75.5% vs. 11.5%) scenarios, though the baseline performs better on additive Gaussian noise.

Significance and Claims
The paper claims that the DHCD benchmark has reached a state of saturation where the "intrinsic floor" of 11 errors is determined by the data quality and resolution (32×32) rather than model capacity. The authors argue that the pursuit of marginal accuracy gains via larger models is futile in this regime. Instead, the significance of Barnamala lies in demonstrating that a highly efficient, parameter-minimal model can achieve statistical parity with massive architectures while offering superior generalization and robustness. The work establishes that for saturated benchmarks, rigorous statistical testing (McNemar) is essential to avoid misleading conclusions based on raw accuracy, and that compact models are preferable for practical deployment due to their transferability and resilience to domain shifts. The authors explicitly state that no configuration escapes the 11-error floor, suggesting that future progress requires new evaluation sets or fundamentally different modeling assumptions.