💬 NLP

BEiTScore: Reference-free Image Captioning Evaluation with an Efficient Cross-Encoder Model

The paper introduces BEiTScore, a reference-free image captioning evaluation metric that leverages a lightweight cross-encoder model trained with adversarial LLM-augmented data to achieve state-of-the-art performance in sensitivity and compositional generalization while maintaining computational efficiency.

Original authors: Gonçalo Gomes, Bruno Martins, Chrysoula Zerva

Published 2026-05-22

📖 4 min read☕ Coffee break read

CC BY 4.0

Original authors: Gonçalo Gomes, Bruno Martins, Chrysoula Zerva

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you are an art critic trying to grade a student's description of a painting. The student says, "A red dog is chasing a blue ball." But in the painting, the dog is actually blue and chasing a red ball.

For a long time, computers trying to grade these descriptions have been like critics who only look at the list of words used, not the story they tell. If the student's list of words matches the painting's list of words (red, dog, blue, ball), the computer gives a high score, even if the description is completely wrong. This is like a teacher giving an "A" to a student who wrote "The blue ball chased the red dog" just because they used the right vocabulary, ignoring that the sentence makes no sense.

This paper introduces a new, smarter grading system called BEiTScore. Here is how it works, broken down simply:

1. The Problem: The "Bag of Words" Trap

Most current computer programs that check image descriptions (like CLIP-based models) treat sentences like a bag of marbles. They count how many "red" marbles or "dog" marbles are in the bag. If the bag has the right marbles, they assume the description is good.

The Flaw: They can't tell the difference between "The dog chased the cat" and "The cat chased the dog." They also struggle with long stories. If a description gets too long (over 77 words), these programs often stop reading and just guess, missing the details at the end.

2. The Solution: A "Detective" Model

The authors built BEiTScore, which acts less like a word-counter and more like a detective.

How it thinks: Instead of just looking at a list of words, it looks at the whole picture and the whole sentence together at the same time. It asks, "Does this specific word fit in this specific spot in this specific image?"
The Training: To teach this detective, the authors didn't just show it correct descriptions. They used a "villain" (a powerful AI) to generate tricky, fake descriptions.
- Example: The AI would take a correct sentence and swap the roles: "The man is holding the woman" becomes "The woman is holding the man."
- The BEiTScore model was trained to spot these subtle tricks, learning to pay attention to who is doing what to whom, not just what objects are present.

3. The "Long Story" Challenge

Imagine trying to read a novel, but your eyes can only focus on the first two sentences before they get tired. That is what older models do with long captions.

BEiTScore's Superpower: It was trained on long, detailed stories about images. It can read the entire caption, from the first word to the last, without getting "tired" or losing focus. It can spot errors that happen deep in the middle or at the very end of a long description, which other models miss.

4. The Results: Smarter and Faster

The paper tested BEiTScore against two types of competitors:

The "Word Counters" (Encoders): These are fast but often make silly mistakes with details.
The "Giant Brains" (LLMs): These are huge, super-smart models that can read long stories and spot details, but they are slow and expensive to run (like using a supercomputer to check a grocery list).

BEiTScore's Achievement:

It caught more mistakes than the "Word Counters."
It was almost as good as the "Giant Brains" at spotting complex errors (like swapping roles or counting objects).
The Best Part: It runs 30 to 100 times faster than the Giant Brains. It's like having a detective who is as sharp as a genius but works at the speed of a regular person.

5. The New Test (LongCapVLCP)

The authors realized that old tests were too easy; they only used short sentences. So, they built a new, harder test called LongCapVLCP.

This test uses very long descriptions and includes tricky errors like text written inside the image (e.g., a sign in the background that says "Open").
On this new, difficult test, BEiTScore proved it could read the long text and spot the errors, while the older "Word Counters" failed completely.

Summary

BEiTScore is a new tool for grading image descriptions. It fixes the old problem of computers only counting words by learning to understand the relationships between words and images. It is smart enough to catch subtle lies in long descriptions, fast enough to be used on thousands of images, and doesn't need expensive, slow super-computers to do the job.

Technical Summary: BEiTScore

Problem Statement

Image captioning evaluation faces significant challenges as Vision-Language Models (VLMs) evolve to generate long-form, context-rich descriptions. Current state-of-the-art evaluation metrics fall into two problematic categories:

LLM-based Judges: While they achieve high correlation with human judgments, they incur prohibitive computational costs and long inference times, making them impractical for large-scale benchmarking or real-time quality-aware decoding.
Encoder-based Metrics (e.g., CLIPScore): These are computationally efficient but suffer from critical limitations. They often treat captions as "bags-of-words," lacking fine-grained sensitivity to compositional relationships. Furthermore, standard CLIP models are biased toward short captions due to pretraining constraints (typically a 77-token limit), leading to poor performance on long-form narratives and an inability to detect subtle visual-linguistic errors.

Existing reference-free metrics struggle to balance the need for high correlation with human judgments, the ability to process long descriptions, and the detection of fine-grained hallucinations (e.g., spatial relations, attribute binding, counting) without relying on expensive LLMs.

Methodology

The authors propose BEiTScore, a new learned, reference-free evaluation metric based on a lightweight cross-encoder architecture.

Model Architecture

Base Model: The metric utilizes the BEiT-3 architecture, a unified multimodal foundation model that represents images through discrete tokens analogous to language tokens.
Initialization: The model is warm-started from a Vision Question Answering (VQA) checkpoint. This provides prior knowledge beneficial for the objective of aligning vision and language.
Output: A regression head (single hidden layer with GeLU activation) processes the cross-modal attention output to produce a score in the range $[0, 1]$ , representing the alignment quality between an image and a caption.

Training Strategy

The training scheme employs a two-stage curriculum using a carefully assembled data mixture designed to expose the model to diverse and complex visual-linguistic relationships:

Stage 1: Pairwise Contrastive Learning
- Objective: Binary discrimination (identifying which of two captions is factually consistent with an image).
- Data: A custom mixture of datasets including Foil-It, TextCaps, Pixel-Prose, and LocalizedNarratives.
- Adversarial Augmentation: To enhance sensitivity to fine-grained errors, the authors employ an adversarial data augmentation pipeline:
  - LLM-based Perturbations: Using Qwen3, they generate factually incorrect but fluent captions by modifying objects, attributes, spatial relations, actions, actant roles, and counting.
  - Synthetic Corruption: Inspired by the "Cobra Effect," they introduce text corruption (word/sentence shuffling, repetition of parts of speech) to counteract bag-of-words behaviors.
  - Hard Negative Mining: Retrieval-based selection of incorrect captions that yield high similarity scores to create easy, medium, and hard negative categories.
- Progressive Training: The training is iterative, starting with foundational reasoning (spatial, roles, actions), followed by quantitative refinement (counting), and finally scene-text integration.
Stage 2: Alignment Fine-tuning
- Objective: Regression to match continuous human judgment scores.
- Data: Training splits from datasets with human ratings, such as VICR, POLOS, Nebula, and EvalMuse-40k.
- Loss: L1 loss is used to optimize direct alignment with normalized human scores ( $q_i \in [0, 1]$ ).

Key Contributions

BEiTScore Metric: A reference-free evaluation metric that leverages a lightweight cross-encoder initialized from a VQA checkpoint. It achieves state-of-the-art performance while remaining significantly more efficient than LLM-based alternatives (20–60× smaller and 30–100× faster).
Adversarial Training Pipeline: A novel data augmentation strategy using LLMs to generate hard negatives and synthetic text corruption, specifically designed to train small models to detect fine-grained visual-linguistic errors and compositional structures.
LongCapVLCP Benchmark: A new benchmark designed to evaluate detailed captioning evaluation beyond the 77-token limit. It includes:
- Long-form Narratives: Assessing performance on captions exceeding standard token limits.
- Scene Text Recognition: A specific task to evaluate the metric's ability to map textual information within an image to its mention in the caption, an aspect underrepresented in prior work.
- Diverse Error Categories: Including actant swaps, action verbs, counting, spatial relations, and scene text errors.

Experimental Results

The authors evaluated BEiTScore across 11 benchmarks, comparing it against encoder-based methods (e.g., CLIPScore, PAC-S, LongCLIP) and LLM-based methods (e.g., EXPERT, FLEUR, VQAScore).

Human Correlation: BEiTScore (specifically the L/14 variant) achieves state-of-the-art Kendall's $\tau$ correlation with human judgments across multiple datasets (VICR, Polaris, Nebula, etc.), often surpassing LLM-based metrics like EXPERT and FLEUR.
Fine-Grained Perception:
- Object Hallucination (nocaps-FOIL): BEiTScore outperforms encoder-based models and matches LLM-based performance in distinguishing correct captions from those with object hallucinations.
- Attribute & Structural Hallucination (SugarCREPE): The model significantly outperforms encoder-based competitors on "swap" tasks, demonstrating superior understanding of structural and relational composition.
- Compositional Reasoning (VALSE & Winoground): BEiTScore excels in tasks requiring holistic reasoning, such as counting, spatial relations, and action understanding, where traditional encoders often fail.
Long-Form and Scene Text (LongCapVLCP):
- Standard encoder-based metrics (CLIP, PAC) perform at or near random chance on captions exceeding 77 tokens or containing scene text errors placed after the token limit.
- BEiTScore maintains discriminative capabilities throughout long narratives and effectively handles scene text recognition, matching the performance of the LLM-based EXPERT metric.
Efficiency: Despite matching or exceeding the performance of specialized LLM-based metrics, BEiTScore requires significantly fewer GFLOPs (e.g., 296 GFLOPs for BEiTScore-L/14 vs. ~9300 GFLOPs for EXPERT).

Significance and Claims

The paper claims that the dominance of LLM-based metrics in image captioning evaluation is not solely a result of model scale but is largely due to the poor exposure of smaller models to complex, long-form textual data during training.

By introducing a targeted training curriculum with adversarial augmentations and utilizing the BEiT-3 architecture, BEiTScore bridges the gap between efficient encoder-based architectures and the complex reasoning capabilities of LLMs. The authors assert that BEiTScore provides a cost-effective solution for automated evaluation, offering strong correlation with human judgments and intricate compositional perception without the computational overhead of large language models. The introduction of the LongCapVLCP benchmark highlights the limitations of current metrics in handling long-form descriptions and scene text, establishing a new standard for evaluating these capabilities.