Towards Annotation-Free Validation of MLLMs: A Vision-Language Logical Consistency Metric

Technical Summary: Towards Annotation-Free Validation of MLLMs: A Vision-Language Logical Consistency Metric

Problem Statement
Current evaluation of Multimodal Large Language Models (MLLMs) relies heavily on accuracy-based metrics derived from ground-truth (gt) annotations. The authors argue that this approach has two critical limitations:

1. Rewarding Unwarranted Guessing: Accuracy metrics may inadvertently reward models that guess correctly without genuine understanding, a phenomenon exacerbated by "mirage reasoning" where models make decisions without utilizing visual inputs.
2. Inapplicability to Novel Tasks: In high-stakes domains or novel tasks where ground-truth annotations are unavailable, traditional accuracy metrics cannot be applied for model validation or selection.

Furthermore, existing self-consistency methods often require building new datasets with specific ground-truth annotations or focus primarily on logical sufficiency (if $p$, then $q$), neglecting logical necessity.

Methodology: Vision-Language Logical Consistency Metric (VL-LCM)
The paper proposes a novel framework based on basic logic principles to evaluate MLLMs on both sufficient and necessary cause-effect relations without requiring ground-truth annotations. The core metric, VL-LCM, is formulated for two distinct benchmark formats:

1. MC-VQA Format (e.g., MMMU, NegBench):

   • Concept: A standard Multiple-Choice (MC) question is decomposed into a set of Yes/No (YN) tests.
   • Logical Formulation:
     • Sufficient Condition: $V, T, A_k \rightarrow a_k$ (If image $V$, question $T$, and choice $A_k$ are present, the answer is "Yes").
     • Necessary Condition: $V, T, \neg A_k \rightarrow \neg a_k$ (If image $V$ and question $T$ are present but choice $A_k$ is absent, the answer is "No").
   • Calculation: The metric computes the joint probability of the model correctly identifying the right choice via MC prediction ($P_{MC}$) and the joint probability of the model correctly validating the choice via YN tests ($P_{JYN}$). The VL-LCM score is the geometric mean of these probabilities, maximizing over all choices.

2. NaturalBench Format:

   • Concept: This format involves pairs of visually similar images and semantically similar questions with alternating answers.
   • Logical Formulation: The framework tests consistency across four derived MC tests and corresponding YN tests, ensuring the model adheres to logical sufficiency and necessity across image-question pairings.
   • Calculation: VL-LCM is computed as the average consistency score across the four sub-tests within a sample unit.

Key Contributions

1. A General Framework: The authors introduce a logic-consistency evaluation framework applicable to both sufficient and necessary conditions, extending beyond traditional self-consistency checks.
2. The VL-LCM Metric: A specific metric defined for typical MC-VQA and NaturalBench formats that operates without ground-truth annotations.
3. Systematic Evaluation: Comprehensive experiments were conducted on five challenging benchmarks (NegBench, ConBench, MMMU, NaturalBench, and a new dataset NatConBench) using 11 recent open-source MLLMs from four frontier families (InternVL, Qwen, LLaVA, Gemma).
4. New Findings on Reliability: The study demonstrates that VL-LCM correlates strongly with F1 scores derived from ground-truth data, validating its utility as an annotation-free proxy for both accuracy and reliability.

Experimental Results

• Accuracy vs. Consistency Gap: While recent MLLMs show significant improvements in accuracy (e.g., InternVL-3.0 reaching ~93% on NegBench), their VL-LCM scores lag significantly (often 20–40% lower). This indicates that high accuracy does not guarantee logical consistency.
• Correlation with Ground Truth: VL-LCM shows strong Pearson and Spearman correlations with ground-truth-based F1 scores and Joint Accuracy (J-Acc), confirming its validity for ranking and selecting models without annotations.
• Response Distribution Analysis: The paper observes a correlation between VL-LCM scores and model response behaviors. Higher VL-LCM scores are associated with reduced "Overconfidence" (hallucinations) and increased "Confidence" in correct answers, suggesting VL-LCM can serve as a reliability indicator.
• Predictive Capability: The metric was shown to predict performance on test sets (where ground truth is hidden) with reasonable accuracy, maintaining ranking orders similar to validation sets.

Significance and Claims
The paper claims that logical consistency, measured by VL-LCM, provides a deeper insight into model performance than accuracy alone. Specifically:

• Annotation-Free Validation: VL-LCM enables the ranking, selection, and validation of MLLMs in novel, real-world tasks where ground-truth data is unavailable.
• Reliability Justification: Beyond model selection, the per-sample nature of VL-LCM allows for the online justification of answers. If a model produces a high VL-LCM score, the answer can be considered more reliable, even without knowing the ground truth.
• Limitations: The authors modestly acknowledge limitations, including the assumption that natural language relations are strictly binary (Yes/No), the increased computational cost of generating additional YN/MC tests, and the need for future work on extending the metric to open-ended natural language responses.

In conclusion, the paper posits that VL-LCM is a necessary complement to accuracy metrics, offering a robust, annotation-free method to assess the logical soundness and reliability of vision-language models.