CaptionQA: Is Your Caption as Useful as the Image Itself?

1. Problem Statement

Image captions are increasingly used as efficient surrogates for visual content in multimodal systems (e.g., retrieval, recommendation, agentic pipelines, and long-term memory). However, current evaluation practices fail to address a fundamental question: Can a generated caption actually stand in for the original image in real-world downstream tasks?

Existing benchmarks suffer from several limitations:

• Traditional Metrics (BLEU, CIDEr): Rely on n-gram overlap with reference captions, ignoring factual correctness and utility.
• Fact-Based Parsers (SPICE, CHAIR): Often rely on complex, non-deterministic pipelines and focus heavily on natural images, lacking depth for specialized domains.
• VLM-as-Judge: Uses another LLM to rate captions based on "thoroughness" or "detail," which often conflates verbosity with utility and introduces non-determinism.
• Image-QA Benchmarks: Evaluate a model's ability to answer questions given the image, not its ability to generate a description that preserves that information.

The core problem is a utility gap: A caption might be grammatically correct and detailed, yet miss the specific visual information required for a downstream agent to make a correct decision.

2. Methodology: The CaptionQA Framework

The authors propose CaptionQA, a utility-based benchmark that evaluates captions by measuring how well they support downstream reasoning tasks without access to the original image.

A. Benchmark Construction

• Domains: The benchmark covers four distinct real-world domains:
  1. Natural: General scenes, objects, spatial relations.
  2. Document: Layouts, tables, charts, and text extraction.
  3. E-commerce: Product attributes, packaging, and marketing details.
  4. Embodied AI: Robotics perception, affordances, and task-relevant cues.
• Taxonomy-Driven: For each domain, human experts and models co-design a fine-grained taxonomy (25 top-level categories, 69 subcategories) defining the specific information needed for downstream tasks.
• Question Generation:
  • Generates 33,027 densely annotated multiple-choice questions (MCQs) across 657 images (avg. 50.3 questions per image).
  • Questions are explicitly tied to taxonomy nodes.
  • Filtering: A text-only LLM filter removes questions answerable by world knowledge alone, ensuring the questions require visual grounding.
  • Deduplication: Embedding-based clustering removes near-duplicate questions.
  • Quality Control: Dual-VLM verification followed by human refinement ensures questions are unambiguous and image-grounded.

B. Evaluation Protocol

The evaluation simulates a real-world pipeline where a downstream system relies solely on text:

1. Caption Generation: A target MLLM generates a caption for an image based on a specific prompt (e.g., "Simple," "Long," "Taxonomy-Hinted").
2. QA-on-Caption: A separate, text-only LLM (Qwen2.5-72B) attempts to answer the benchmark MCQs using only the generated caption as input.
3. Scoring:
   • Accuracy (Acc): Fraction of questions answered correctly.
   • Cannot Ratio: Fraction of questions where the QA model correctly identifies it cannot answer (indicating missing info).
   • Utility Score: A weighted score that rewards correct answers, penalizes hallucinations (wrong answers), and gives partial credit for "Cannot answer" to distinguish between missing info and hallucination.

3. Key Contributions

1. CaptionQA Benchmark: The first utility-oriented benchmark covering 4 diverse domains with 33k+ questions and a structured taxonomy (25 top-level, 69 subcategories).
2. Utility-Centric Protocol: A deterministic, low-cost evaluation method that directly measures if a caption preserves image-level utility for downstream LLMs, moving beyond "how good the caption sounds" to "what the caption enables."
3. Open-Source Pipeline: The authors release the full construction pipeline, allowing the community to easily extend CaptionQA to new domains by simply defining a new taxonomy and curating images.

4. Key Results

A. The Utility Gap

There is a substantial performance drop when moving from QA-on-Image (model sees the image) to QA-on-Caption (model sees only the caption):

• Proprietary Models: Drop by 9.2% – 16.4%.
• Open-Source Models: Drop by 11% – 32.4%.
• Embodied AI: The gap is most severe here, exceeding 40% for some models, indicating that current captions fail to capture critical spatial and task-relevant cues for robotics.

B. Model Performance Discrepancies

• Models that perform similarly on traditional Image-QA benchmarks (e.g., MMBench) can diverge significantly on CaptionQA.
  • Example: Claude Sonnet 4.5 and LLaVA-OneVision-7B differ by only ~1% on Image-QA but by 17.2% on CaptionQA.
  • Example: GPT-5 and LLaVA-OneVision-7B differ by ~9% on Image-QA but by 32.3% on CaptionQA.
• Proprietary models (GPT-5, Gemini 2.5 Pro) generally lead, but even they lose significant utility (e.g., GPT-5 drops from ~98% Image-QA to ~89% CaptionQA).

C. Prompt Analysis

• Length vs. Utility: Increasing caption length from "Short" to "Simple" yields massive gains (+33.8%). However, going from "Simple" to "Long" offers diminishing returns (+0.35%), suggesting that verbosity does not equal utility.
• Complex Prompts Backfire: Using "Taxonomy-Hinted" prompts (explicitly listing categories) often degrades performance (mean -10.8%). Models tend to imitate the format (fill-in-the-blank style) rather than providing grounded descriptions, leading to hallucinations or superficial coverage.
• Recommendation: The "Simple" prompt ("Describe this image in detail") achieves 99% of the benefits of "Long" prompts at a fraction of the token cost.

D. Domain-Specific Insights

• E-commerce: Easiest domain (smallest gap), as product attributes are often textual and structured.
• Embodied AI: Hardest domain. Captions frequently miss sensor geometry, depth, and precise spatial relationships required for robotic manipulation.
• Document: Struggles with structural elements (tables, charts) where layout is critical but hard to describe textually.

5. Significance and Impact

• Redefining Evaluation: Shifts the paradigm from "caption quality" (fluency/overlap) to "caption utility" (downstream task success).
• Practical Deployment: Provides a concrete metric for engineers deploying multimodal systems. If a caption causes a downstream agent to fail 30% more often, the captioning model is unsuitable for that pipeline, regardless of its Image-QA score.
• Guidance for Model Development: Highlights that current MLLMs are excellent at "seeing" but poor at "summarizing for action." The gap in Embodied AI suggests a need for models specifically trained to capture affordances and spatial reasoning in text.
• Community Resource: By open-sourcing the pipeline, the authors enable rapid adaptation of this evaluation standard to emerging domains (e.g., medical imaging, legal documents) without rebuilding the benchmark from scratch.

In conclusion, CaptionQA reveals that current state-of-the-art MLLMs generate captions that are often insufficient for real-world applications, acting as a "lossy compression" of visual information that can lead to significant errors in downstream decision-making.