💬 NLP

Adjudicated Captioning: Multi-Agent Alignment Scoring and Consensus-Distilled Beam Arbitration for Strict Zero-Shot Image Captioning

The paper proposes "Adjudicated Captioning," a training-free, multi-agent inference framework that enhances strict zero-shot image captioning by integrating frozen retrieval encoders and cross-attention verifiers with self-supervised, consensus-distilled rerankers to achieve state-of-the-art performance on COCO and Flickr30k without retraining the underlying captioner.

Original authors: Duy Tran Thanh, Thien-Phuc Doan, Long Nguyen-Vu, Ngo Tan Vu Khanh

Published 2026-08-03

📖 6 min read🧠 Deep dive

CC BY 4.0

Original authors: Duy Tran Thanh, Thien-Phuc Doan, Long Nguyen-Vu, Ngo Tan Vu Khanh

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are trying to teach a robot to describe a picture it has never seen before. You can't show it the picture and the correct description together during its school years; that would be "violating the strict zero-shot learning protocol" in the world of strict zero-shot learning. Instead, you give the robot a massive library of text descriptions and a special pair of glasses that can roughly guess what a picture looks like based on words. The robot's job is to look at a new photo, find the most similar descriptions in its library, and then use those to write a new caption.

For a long time, this process hit a frustrating wall. The robot would find a few similar descriptions, pick the best ones, and then just let its internal "language brain" finish the sentence. The problem was that once the robot started writing, it stopped checking if what it was saying actually matched the picture. It became too focused on sounding fluent and grammatically correct, often ignoring the visual details. It was like a tour guide who memorized a script about a famous mountain but, once the tour started, just kept talking about how beautiful mountains are in general, forgetting to mention that the specific mountain in front of them was actually a volcano.

This paper tackles that exact problem. The researchers, working in the field of artificial intelligence and computer vision, propose a new way to keep the robot honest. They call their method "Adjudicated Captioning." Instead of letting the robot write the whole story and then hoping for the best, they install a team of strict "judges" who check the work at multiple stages. They don't retrain the robot's brain (which would require seeing the pictures and answers together); instead, they add a smart, self-correcting layer that runs after the robot has done its initial draft.

Here is how their new system works, using a simple story:

The Old Way: The One-Time Check
In the old system, the robot looked at a photo of a snowboarder and asked its library, "What words are close to this?" It found nine similar descriptions. It picked the top five, and then its language brain started writing. It wrote, "A snowboarder and a snowboarder are posing." The system stopped there. It never asked, "Wait, does the picture actually show two people?" It just trusted its language brain to be right. This led to a score of 108.0 on a standard test called CIDEr (a way to measure how good a caption is).

The New Way: The Multi-Stage Courtroom
The authors realized the robot needed more than one check. They built a pipeline with three new "judges" who work together without ever seeing the "correct" answers (the ground truth).

The Stronger Librarian (Stage 1): First, they swapped the robot's glasses for a much sharper pair. Instead of a standard lens, they used a massive, powerful model (OpenCLIP ViT-bigG/14) to find the initial descriptions. This alone made the robot smarter, bumping the score up to 111.6.
The Cross-Examiner (Stage 2): Next, they added a second judge. This judge doesn't just look at the words; it uses a special "cross-attention" tool (a BLIP-ITM Verifier) to really study the relationship between the photo and the text. It takes the top nine descriptions found by the librarian and re-ranks them, picking the best five. This judge is different from the librarian; it catches things the librarian misses. If the librarian thinks a photo is about "a man," this judge might realize it's actually "two men." This step is crucial because the two judges often disagree, and that disagreement helps the system find the truth.
The Final Arbitrator (Stage 4): Finally, the robot generates a list of 20 possible sentences (a "beam"). Usually, it would just pick the one that sounds the most fluent. But now, two new, tiny, self-taught AI heads (named TriFuse and MemAttend) step in. These heads look at the 20 sentences and ask three questions for each one:
- Does the language brain like it?
- Does the Stronger Librarian think it matches the photo?
- Does the Cross-Examiner think it matches the photo?

These two new heads are trained in a clever way. They don't need a teacher with the right answers. Instead, they look at the three judges above. If all three judges agree that a specific sentence is the best, the new heads learn to pick that sentence. It's like a student learning by watching a panel of experts agree on the right answer, rather than being told the answer directly.

The Result
By adding these checks, the system didn't just sound better; it became more accurate. On the COCO Karpathy test, the new system reached a CIDEr score of 117.6, a massive jump from the previous best of 108.0. Even more impressively, it beat a method that used "fake" images generated by other AI to train itself (which scored 109.9), all without the robot ever seeing a single paired image and caption during its training.

The paper also tested this on different types of photos, like those from Flickr30k and NoCaps. The system improved those scores too, gaining +8.1 and +5.7 points respectively, proving that this "multi-judge" approach works even on pictures the robot wasn't specifically trained on.

What They Ruled Out
The researchers were careful to show that this wasn't magic. They proved that simply making the language brain stronger wasn't the answer; the improvement came from where and how they checked the image-text match. They also showed that just adding a second judge wasn't enough; the judges had to be different types of models (one a "dual encoder" and one a "cross-attention matcher") so they could catch different kinds of mistakes. They even tested a "negative" idea: trying to use the worst descriptions to teach the robot what not to say. That actually made things worse, proving that focusing on the best matches was the right path.

How Sure Are They?
The paper is very confident in these numbers. They ran the tests multiple times and found the results were stable, with only tiny, random fluctuations (about 0.1 points) due to computer hardware quirks. They also showed that the improvement wasn't just because they happened to use a specific version of a judge trained on the exact same test set; even when they swapped in a judge trained on a different set of photos, the system still improved significantly.

In short, this paper suggests that the reason AI captioning had stalled wasn't because the robots were too dumb, but because they were too trusting of their own words. By adding a team of independent, self-supervised judges to verify the work at every step, the system learned to describe the world more accurately, all without breaking the rules of strict zero-shot learning.

Technical Summary: Adjudicated Captioning

Problem Statement
The paper addresses the stagnation in Strict Zero-Shot Image Captioning (ZIC), a regime where the Captioner (the text generator) is trained exclusively on text corpora without paired image-caption supervision. Existing retrieval-augmented methods in this regime, such as IFCap, suffer from a structural bottleneck: they score image-text alignment exactly once at the initial retrieval step using a dual-encoder (typically CLIP ViT-B/32). Once the retrieval pool is established, the Captioner's autoregressive beam is selected solely based on language-model probability. This leaves the decoder without further visual grounding feedback, resulting in fluent but potentially ungrounded outputs. Progress in this strict regime has plateaued since 2024, with no method improving upon IFCap's CIDEr score of 108.0 on the COCO Karpathy benchmark without resorting to synthetic image augmentation or retraining the Captioner.

Methodology: Adjudicated Captioning
The authors propose Adjudicated Captioning, an inference-time multi-agent framework that restores visual grounding feedback at multiple checkpoints without modifying the underlying Captioner (IFCap). The pipeline introduces three novel components atop the unchanged IFCap architecture:

Stronger Retrieval Encoder (Stage 1): The initial retrieval step is upgraded from OpenAI CLIP ViT-B/32 to a scaling-law-stronger OpenCLIP ViT-bigG/14. This retrieves the top-9 candidate captions from the COCO training corpus based on cosine similarity.
Cross-Attention Verifier (Stage 2): A frozen BLIP-ITM Cross-Attention Verifier is inserted between retrieval and decoding. It re-ranks the top-9 candidates down to a top-5 "Verifier-adjudicated" pool. This component provides orthogonal information to the dual-encoder retrieval, as evidenced by a low rank correlation ( $\rho \approx 0.4$ ) between the CLIP cosine scores and the Verifier's matching scores.
Learned Beam Rerankers (Stage 4): At the output, two complementary, self-supervised learned modules select the final caption from the Captioner's 20-wide beam:
- TriFuse: A minimal 113-parameter MLP that fuses three signals per candidate: the Language Model (GPT-2) log-probability, the Retrieval Encoder cosine, and the Verifier's matching score.
- MemAttend: A 17.4K-parameter memory-augmented transformer encoder that allows beam candidates to cross-attend to the Verifier-adjudicated retrieval pool (memory tokens).

Training Paradigm
Crucially, the learned components (TriFuse and MemAttend) are trained strictly self-supervised under the captioner-text-only regime. They do not use any paired image-caption labels or reference captions. Instead, they are trained via Borda-consensus distillation across the three frozen scorers (LM, Retrieval, Verifier). The pseudo-label for each candidate is derived from the average inverted rank of the three scorers, and the models are trained to minimize listwise cross-entropy against this consensus target.

Key Contributions

Architectural Hypothesis: The paper identifies the single-checkpoint alignment scoring as the cause of the ZIC plateau and proposes a remedy: placing strong, frozen image-text scorers at multiple checkpoints (retrieval, verification, and output).
Self-Supervised Rerankers: The introduction of TriFuse and MemAttend as the first learned Beam Rerankers proposed specifically for the captioner-text-only ZIC regime.
Performance Gains: The framework achieves a CIDEr of 117.6 and SPICE of 21.9 on COCO Karpathy, surpassing the previous strict-ZIC state-of-the-art (IFCap, 108.0) by +9.6 CIDEr and the strongest synthetic-augmented method (NES, 109.9) by +7.7 CIDEr, all without retraining the Captioner.
Cross-Domain Transfer: The inference-only recipe transfers effectively to Flickr30k (+8.1 CIDEr) and NoCaps (+5.7 CIDEr) without Captioner retraining.

Results and Ablation

Decomposition of Gain: The total +9.6 CIDEr gain is decomposed as follows:
- +7.8 CIDEr is attributed to non-learned architectural interventions (stronger Retrieval Encoder, Verifier rerank, and entity-filter threshold rescaling). A training-free fixed-fusion baseline achieves 115.8 CIDEr.
- +1.8 CIDEr is attributed to the learned TriFuse+MemAttend ensemble.
Hallucination Reduction: The framework reduces sentence-level reference-proxy hallucination (CHAIRs) by approximately 41% (from 13.28% to 7.84%), indicating more grounded descriptions.
Verifier Substrate: Swapping the COCO-fine-tuned Verifier for a Flickr30k-fine-tuned variant still yields a +6.0 CIDEr lift over IFCap, confirming that the architectural intervention is the primary driver of performance, not just COCO-specific tuning.

Significance and Claims
The paper claims that the strict-ZIC plateau is an architectural artifact rather than a fundamental limit of language modeling. By moving from a single-checkpoint alignment strategy to a multi-checkpoint "adjudicated" pipeline, the authors demonstrate that retrieval-augmented generation can be significantly improved without paired supervision or Captioner retraining. The work establishes that placing a cross-attention verifier between retrieval and decoding, followed by a consensus-distilled reranker, effectively recovers alignment signals lost in traditional pipelines. The authors position their method as a new operational baseline for strict-ZIC, outperforming even some paired-supervised baselines (like ClipCap) on CIDEr and SPICE, while maintaining the strict text-only training constraint.

Technical Summary: Adjudicated Captioning

More like this