💻 computer science

HAT Super-Resolution and a PARSeq+CLIP4STR Voting Ensemble for Extreme In-the-Wild License Plate Recognition

This paper presents a license plate recognition system for the ICIP 2026 XLPSR challenge that combines HAT super-resolution with a PARSeq and CLIP4STR voting ensemble to achieve a 9.73 wECR score while adhering to strict inference time constraints.

Original authors: Karthik Sivarama Krishnan, Koushik Sivarama Krishnan

Published 2026-07-13

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Karthik Sivarama Krishnan, Koushik Sivarama Krishnan

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you're trying to read a license plate on a car zooming past you in a storm. The plate is tiny—sometimes just 12 pixels wide (that's smaller than a single fingernail on your screen)—and it's blurry, squashed by JPEG compression, and maybe even half-hidden. Trying to read it is like trying to guess the letters on a postage stamp that's been shrunk down to the size of a dust mote.

This paper describes a team's entry into a high-stakes contest called the "Extreme In-the-Wild License Plate Super-Resolution" challenge. Their goal wasn't just to guess the letters; it was to figure out how to make the blurry mess readable enough to guess correctly, while avoiding the penalty for guessing wrong.

The Core Discovery: Make the Pixels Bigger First

The team's biggest "Aha!" moment was realizing that the most powerful tool wasn't a smarter brain, but better eyes. They found that Super-Resolution (making the image bigger and sharper) was the single most important step.

Think of it like this: If you have a tiny, blurry drawing of a letter "A" that is only 2 or 3 pixels wide, no amount of studying will help you tell if it's an "A" or an "H." The signal is too weak. But if you use a magic magnifying glass (their HAT system) to blow that image up 4 times, suddenly the strokes become thick and clear. The paper shows that adding this "magnifying glass" boosted their score by a massive +2.00 points on their scoring scale. Without it, the system was basically guessing in the dark.

The Team of Detectives

Once the image was magnified, the team didn't just use one detective to read the plate. They used a team of two:

PARSeq-S: A fast, efficient reader that looks at the image in a specific, organized way.
CLIP4STR-B: A heavier, more powerful reader that was trained on a huge library of image-text pairs.

They treated these two like a voting committee. When they both looked at a letter, they didn't just take a simple majority. They weighed the votes: PARSeq got 2 votes, and CLIP4STR got 1 vote. Why? Because in their testing, this specific mix worked better than giving them equal say.

The "Don't Guess" Rule

Here is where the game gets tricky. The contest rules were harsh:

Get a letter right: +2 points.
Get a letter wrong: -1 point.
Don't guess (leave it blank): 0 points.

This means if you are less than 33% sure about a letter, guessing is actually a bad idea because you're likely to lose more points than you gain. The team built a clever "safety valve" into their system. If the computer's confidence score for a letter dropped below 0.33 (33%), the system would simply abstain—it would say, "I don't know," and leave a blank space instead of guessing.

This strategy turned potential mistakes into safe zeros, adding another +0.11 points to their final score. It's like a quiz where you only answer the questions you're sure about, rather than panicking and filling in bubbles randomly.

What They Rejected (The "No-Go" List)

The team was very careful about what not to do, and they proved these ideas didn't work:

No "More is Better" for Models: They tried adding a third detective (a model called SVTRv2) to the team. It didn't help; it actually made things worse. The paper suggests that adding a model that thinks differently (using a different "decoder" style) just created more noise and confusion, not more correct answers.
No Rigid Rules: They tried forcing the system to strictly follow French license plate rules (like never using the letter 'I' or 'O' because they look like numbers). This backfired, lowering their score. Sometimes the strict rules filtered out correct answers that looked weird but were actually right.
No Cheating: They didn't use any secret data or manual help. They only used the official training data and public pre-trained models.

The Final Score and Speed

By combining the "magic magnifying glass" (Super-Resolution), the weighted voting team, and the "don't guess" safety rule, they reached a score of 9.73 on a scale where 10 is perfect.

They also checked how fast this was. On a powerful graphics card (an RTX 3090), the whole process took about 1.7 seconds per car sequence. The contest allowed up to 60 seconds, so they had plenty of time to spare.

In short, the paper proves that for these tiny, blurry plates, making the image readable is more important than having a super-complex brain. You can't read what you can't see, and once you can see it clearly, a smart, cautious team of readers can do the rest.

Technical Summary: HAT Super-Resolution and PARSeq+CLIP4STR Voting Ensemble for Extreme In-the-Wild License Plate Recognition

Problem Statement
This work addresses the ICIP 2026 Grand Challenge on Extreme In-the-Wild License Plate Super-Resolution (XLPSR). The task involves predicting the text of French license plates from video sequences where the plates suffer from extreme degradation: widths as narrow as 12 pixels, motion blur, JPEG compression, adverse lighting, and occlusion. The evaluation metric is a per-character scoring rule (+2 for correct, −1 for incorrect, 0 for abstention). This asymmetric scoring creates a "negative expected value" scenario for guessing in low-confidence regions, as the probability of a correct guess must exceed 1/3 to be beneficial. The challenge provides 10 frames per sequence, requiring a system that can recover legibility from sub-pixel character strokes.

Methodology
The proposed pipeline treats the problem as a recognition task gated by image legibility, prioritizing the recovery of stroke information over raw throughput. The system operates in four distinct stages:

Multi-Frame Fusion:
- From the 10 input frames, the system ranks crops by Laplacian variance to identify the sharpest reference frame.
- Remaining frames are aligned to this reference using the Enhanced Correlation Coefficient (ECC) algorithm under an affine motion model. ECC is selected for its robustness to low-contrast conditions and lack of need for ground-truth flow supervision.
- The aligned frames are averaged pixel-wise to create a fused crop.
- The pipeline retains six views for processing: the sharpest reference, up to four additional aligned crops (in temporal order), and the fused crop.
Super-Resolution (SR):
- Each of the six views is upscaled 4× using HAT (Hybrid Attention Transformer), specifically the Real-HAT-GAN-SRx4 variant.
- The model utilizes six RHAG blocks with an embedding dimension of 180, a window size of 16, and a PixelShuffle upsampler.
- The authors use public weights without further fine-tuning, noting that the 60-second per-sequence budget allows for prioritizing legibility over the speed of lighter real-time variants.
Ensemble Recognition:
- The super-resolved views and the original-resolution reference are passed through two distinct scene-text recognizers:
  - PARSeq-S v2: A 23.8M parameter ViT-Small with a permuted autoregressive decoder, processing 32×128 inputs.
  - CLIP4STR-B: A 158M parameter model using a CLIP-ViT-B/16 backbone with a cross-modal decoder, processing 224×224 inputs.
- Both models are fine-tuned on synthetic French plate data (generated via OpenCV and FE-Schrift fonts with BSRGAN-style degradation) and the XLPSR development set.
- This ensemble provides diversity in backbone architecture, input resolution, and pre-training corpora (ImageNet-1K vs. CLIP's 400M image-text pairs).
Confidence-Weighted Voting with Abstention:
- Predictions are filtered to retain the modal string length (preferring 7 characters, the French SIV format).
- For each character position, soft-max confidences are aggregated across all retained tuples. The voting scheme weights PARSeq twice and CLIP4STR once (2:1 ratio), determined via ablation.
- Abstention Mechanism: If the winning class's soft-max mass falls below a threshold of $\tau = 0.33$ , the system abstains (outputs _). This threshold aligns with the +2/−1/0 scoring breakeven point ($P(correct) = 1/3$).
- A post-processing step applies SIV-format corrections (mapping I→J, O→D, U→V) but avoids hard format constraints during the voting phase, as they were found to filter out correct edge-case predictions.

Key Results
The system achieved a 9.73 wECR on the public validation leaderboard. An ablation study (Table I) highlights the following contributions to performance:

Super-Resolution Dominance: The SR step contributed the largest gain (+2.00 wECR) over a baseline without SR. At extreme scales (12–20 px plate widths), characters occupy only 2–3 pixels; HAT lifts these strokes above the sub-pixel threshold where OCR backbones typically fail.
Ensemble Diversity: Combining PARSeq and CLIP4STR added +0.31 wECR over PARSeq alone. However, adding a third model with a different decoder family (CTC-based SVTRv2) degraded performance, suggesting that increasing disagreement without increasing correlated correctness introduces vote noise.
Abstention Strategy: The confidence-weighted voting with $\tau=0.33$ abstention improved the score by +0.11 wECR by converting likely incorrect guesses (−1) into neutral abstentions (0).
Efficiency: The full pipeline runs in approximately 1.7 seconds per sequence on an RTX 3090 (p99 latency 2.4s), well within the 60-second limit.

Significance and Claims
The paper claims that for extreme-scale license plate recognition, legibility recovery is the binding constraint, not decoder capacity. The authors demonstrate that a carefully engineered super-resolution front-end, combined with modest ensemble diversity and an abstention strategy calibrated to the specific asymmetric scoring rule, is sufficient to achieve high performance without proprietary models or external data. The work explicitly rejects the notion that larger models or complex decoder families are the primary solution to sub-pixel recognition challenges, instead emphasizing the necessity of lifting image quality to a recognizable threshold before recognition occurs.