💻 computer science

Spike-HTR: Spiking Neural Transformer for Handwritten Text Recognition

Spike-HTR is a hybrid spiking neural network for handwritten text recognition that addresses the computational mismatch between static images and spiking dynamics by employing an InkCoder for coarse-to-fine temporal encoding and a CTC-guided length reducer to compress blank-dominated sequences, achieving high accuracy with only two timesteps and without external language models.

Original authors: Xiubo Liang, Jinxing Han, Yuke Li, Haoqi Zhu, Yu Zhao, Hongzhi Wang

Published 2026-08-04

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Xiubo Liang, Jinxing Han, Yuke Li, Haoqi Zhu, Yu Zhao, Hongzhi Wang

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to read your messy handwriting. Usually, the robot acts like a super-organized librarian who checks every single inch of the page, even the empty white spaces between your words. It spends a lot of energy and time staring at blank paper, just to be sure it hasn't missed a tiny dot. This is how most modern computer vision works: it processes the whole image at once, regardless of whether the ink is there or not.

But there's a different kind of computer brain called a Spiking Neural Network (SNN). Think of an SNN not as a librarian, but as a nervous system. It doesn't "think" in a steady stream; it fires little electrical sparks, or "spikes," only when something interesting happens. If there's no ink, there's no spark. This makes SNNs incredibly efficient, like a flashlight that only turns on when you point it at something. However, there's a catch: handwriting is a static picture (a frozen moment in time), but SNNs are designed to work over time, like a movie. Trying to feed a frozen photo into a time-based brain is like trying to play a still photograph on a movie projector; the brain gets confused because nothing is moving, so it either wastes energy repeating the same image or gets jittery trying to guess when to fire.

This is the puzzle that the researchers behind Spike-HTR are trying to solve. They wanted to build a handwriting reader that uses these efficient, spark-firing brains without losing accuracy. Their main finding is that they can make this work by being very smart about when and where the robot looks. They discovered that by carefully controlling two things—how many "moments" the robot spends looking at the image, and how many parts of the page it actually processes—they can get the robot to read handwriting almost as well as the heavy, energy-hungry models, but with a much lighter footprint.

The paper suggests that the secret isn't just in the brain itself, but in how you feed it the information. The team created a new system called InkCoder. Imagine you are describing a painting to a friend over the phone. Instead of saying "Here is the whole picture" and repeating it five times, you start by saying, "Okay, imagine a big red blob on the left," and then, "Now, zoom in, there's a sharp line inside that blob." That is exactly what InkCoder does. It takes a single, static image of handwriting and turns it into a short, 2-step "movie" for the spiking brain. The first step shows the broad strokes (the big blobs), and the second step zooms in on the sharp edges and details. This way, the brain doesn't waste time replaying the same static image; it uses each moment to refine its understanding, moving from a rough sketch to a sharp picture.

But there's a second trick up their sleeve. Handwriting often has long stretches of empty space between words. The researchers noticed that their "deep mixer" (the part of the brain that connects the dots to form words) was wasting energy processing these blank gaps. So, they added a CTC-guided length reducer. Think of this as a smart editor who scans the page before the deep reading begins. If the editor sees a long, empty gap, they say, "We don't need to read this part in detail; let's just skip it." However, they are careful not to skip the tiny gaps that separate two similar letters (like the space between two 'i's). This tool compresses the long, blank stretches into a tiny space, keeping only the parts where the ink actually is or where the robot is unsure.

The results are quite promising. When they tested this system on three different handwriting datasets (including English, Italian, and old German manuscripts), the model performed very well. With just 2 time steps (T=2), it achieved error rates of 5.4% on English, 2.5% on Italian, and 3.9% on German. These numbers are competitive with much larger, traditional models, but the key difference is efficiency. The paper shows that the system is "sparse," meaning it fires very few sparks overall. Most of the activity happens in the early layers where the image is big, and as the information gets processed, the sparks become even more focused.

The authors are careful to note that this isn't a magic bullet that solves everything yet. They found that adding more time steps (like going from 2 to 4) didn't help much more; the system seemed to hit a wall where extra time didn't make it smarter, just a bit more precise with word boundaries. They also point out that while the "spiking" part is efficient, the early parts of the system still use some traditional, non-spiking math to make sure the image is clear. This is a trade-off: they kept the early layers "dense" to protect the delicate details of the handwriting, which means the system isn't 100% spark-based yet.

In short, Spike-HTR suggests that to make efficient handwriting readers, we shouldn't just build better brains; we need to build better ways to feed them information. By turning a static photo into a short, refining story (InkCoder) and cutting out the boring blank spaces (the length reducer), we can get these efficient, spark-firing computers to read our messy notes without burning a hole in the power grid. It's a step toward making smart devices that can read our handwriting on the fly, using a fraction of the energy current computers need.

Technical Summary: Spike-HTR: Spiking Neural Transformer for Handwritten Text Recognition

1. Problem Statement

Handwritten Text Recognition (HTR) faces a fundamental computational imbalance:

Spatial Redundancy: Most pixels in a handwriting image are background, yet standard recognizers (CNNs, Transformers) process the full canvas with dense operations.
Sequential Redundancy: When 2D features are collapsed into a 1D sequence along the width axis, many positions correspond to blank spaces or margins. In Connectionist Temporal Classification (CTC) frameworks, these are often assigned blank labels, yet standard deep sequence mixers still allocate equal computational budget to these blank-dominated spans.

This creates a specific mismatch for Spiking Neural Networks (SNNs). SNNs rely on event-based, sparse computation unfolding over time. However, offline HTR inputs are static images. Common workarounds (repeating the image or stochastic Poisson sampling) are either information-redundant or introduce noise that perturbs thin strokes, making short-horizon spiking inference ineffective.

2. Methodology: Spike-HTR

The authors propose Spike-HTR, a hybrid spiking recognizer that explicitly manages two computational budgets to align with the sparse nature of handwriting evidence:

Temporal Budget ( $T$ ): The number of spiking timesteps.
Sequence Budget ( $\ell_b$ ): The number of width-axis positions processed by the deep 1D mixer after length reduction.

2.1. InkCoder: Deterministic Coarse-to-Fine Temporal Entry

To resolve the static-to-temporal bottleneck, the paper introduces InkCoder, a deterministic input module that converts a single static image into a short sequence of spatial gates $\{g_t\}_{t=1}^T$ .

Mechanism: InkCoder constructs two complementary evidence streams from the grayscale image:
1. Intensity Stream: Estimates stable stroke mass, robust to illumination and thickness variations.
2. Edge Stream: Recovers thin strokes and junction topology, filtered by cross-modal consistency (trusting edges only where intensity indicates ink) and spatial coherence (suppressing isolated texture noise).
Temporal Fusion: These streams are fused via a step-dependent convex combination. Early timesteps ( $t=1$ ) emphasize the stable intensity stream to capture broad stroke regions. Later timesteps progressively increase the weight of the edge stream to sharpen boundary details.
Gating: The fused evidence is stabilized by density-based speckle suppression and converted into spatial gates via scheduled soft-thresholding. These gates modulate the drive injected into the spiking encoder, ensuring timesteps are used for structured refinement rather than redundant replay.

2.2. Spiking Encoder

The backbone utilizes a lightweight non-spiking stem to extract a shared spatial feature map, which is then processed by a multi-step spiking 2D encoder.

Architecture: A three-stage hierarchy with stride-2 transitions.
Preservation Mechanisms: To counteract low-pass integration inherent in spiking neurons, the model employs:
- Membrane Shortcuts: Analog pre-activations are added to spiking outputs to preserve fine stroke topology.
- Dual-Resolution Fusion: High-resolution details from Stage 2 are injected into Stage 3 via content-conditioned gates.
Pooling: The final 2D map is collapsed into a 1D token sequence via height pooling, resulting in a nominal length $L_0 \approx W/4$ .

2.3. CTC-Guided Length Reducer

Before the deep 1D mixer, a CTC-guided length reducer compresses the sequence from $L_0$ to $\ell_b$ .

Preview Mechanism: A stop-gradient preview is computed using a shared CTC head on the fused spiking features. This estimates the blank posterior probability ( $p_{blank}$ ) and uncertainty ( $u$ ) for each position.
Keep-and-Merge Strategy:
- Keep: Positions with low blank probability (likely characters) or high uncertainty are retained.
- Merge: Long stretches of likely blank positions are compressed into short spans (up to size $k$ ) via blank-aware weighted pooling.
- Constraint: A minimum keep ratio ( $\gamma$ ) is enforced to ensure feasibility for separating repeated characters.
Scope: This component is SNN-agnostic and can be applied to ANN-CTC recognizers as well.

2.4. 1D Sequence Mixer and Temporal Fusion

The shortened sequence is processed by a stack of hybrid 1D mixer blocks containing linear attention, QK-style mixing, and large-kernel spiking convolutions. Finally, the $T$ spiking steps are fused via a learned convex combination to produce the final CTC logits.

3. Key Contributions

Two-Budget Formulation: The paper frames static-image spiking HTR as a problem of allocating limited temporal ( $T$ ) and sequence ( $\ell_b$ ) budgets to sparse evidence.
InkCoder: A novel input module that transforms static handwriting into a meaningful, short, coarse-to-fine temporal stream, enabling effective low-timestep ( $T=2$ ) spiking inference without stochastic noise or redundancy.
CTC-Guided Reducer: An order-preserving length reduction mechanism that compresses blank-dominated sequences while retaining critical character and uncertain positions, reducing the computational load of the deep mixer.
Spike-HTR Model: A spiking Transformer trained from scratch (no external pretraining or synthetic data) that achieves competitive accuracy with minimal timesteps.

4. Experimental Results

The model was evaluated on three benchmarks: IAM (English), LAM (Italian), and READ2016 (German). All models were trained from scratch without external language models or lexicons, using greedy CTC decoding.

Accuracy: With $T=2$ $T = 2$ , Spike-HTR achieved Character Error Rates (CER) of:
- IAM: 5.4% (Test)
- LAM: 2.5% (Test)
- READ2016: 3.9% (Test)
Temporal Scaling: Increasing $T$ beyond 2 yielded diminishing returns. While CER saturated at $T=2$ , Word Error Rate (WER) saw slight improvements up to $T=4$ , suggesting extra timesteps refine word boundaries rather than character substitutions.
Ablation Studies:
- InkCoder: Replacing InkCoder with repetition, Poisson coding, or TTFS resulted in significant performance drops (e.g., IAM CER rose from 3.5% to 4.3% with repetition).
- Reducer: The keep-and-merge strategy reduced the mixing sequence length by ~32% with negligible accuracy loss compared to no reduction.
- ANN Control: An isomorphic ANN (replacing spiking neurons with GELU) performed slightly worse (3.558% vs 3.489% CER on IAM validation), suggesting a small benefit from spiking dynamics under strict budget constraints.
Activity Diagnostics: The model exhibited globally sparse activity. While the 1D mixer had higher firing rates, the 2D encoder dominated the total spike volume due to larger activation maps.

5. Significance and Claims

The paper claims that Spike-HTR demonstrates that offline HTR with SNNs is not merely a backbone design challenge but a budget allocation problem.

Efficiency: By controlling $T$ and $\ell_b$ , the model aligns computation with the actual density of stroke evidence. The length reducer offers a general mechanism to reduce compute for CTC-based recognizers, while InkCoder specifically addresses the static-to-temporal entry problem for SNNs.
Hardware Relevance: The authors note that on current dense accelerators, the primary immediate benefit is the reduced sequence length for the deep mixer. However, the sparse firing profiles (quantified via spike-share and firing rates) indicate that the model preserves the event-driven sparsity required for future neuromorphic deployment, where data movement and activity are the primary energy constraints.
Modesty: The authors acknowledge limitations, including the hybrid nature of the early visual pathway (dense stem + analog shortcuts) which limits "spike-native" execution, and the reliance on specific priors (intensity/edge coherence) that may struggle with extreme degradation or non-Latin scripts. They emphasize that the reported efficiency gains are currently best viewed as algorithmic diagnostics rather than immediate wall-clock speedups on standard GPUs.