LLM-as-Judge Framework for Evaluating Tone-Induced Hallucination in Vision-Language Models

1. Problem Statement

Vision-Language Models (VLMs) are increasingly deployed in high-stakes environments (e.g., medical imaging, security) where reliable visual grounding is critical. However, current research has insufficiently characterized how VLMs behave under progressively coercive prompt phrasing.

Existing hallucination benchmarks suffer from three primary limitations:

1. Binary Prompt Manipulation: They typically treat prompts as either "adversarial" or "benign," ignoring the graded continuum of linguistic pressure (from neutral inquiry to coercive demand) that exists in real-world interactions.
2. Binary Evaluation: They focus on whether a hallucination occurs, failing to measure the intensity, confidence, or specificity of the fabrication. A hedged guess and an elaborate lie are often treated identically.
3. Task Narrowness: Benchmarks focus heavily on object presence/absence, neglecting finer-grained constraints like illegible text or unreadable temporal displays.
4. Lack of Ground-Truth Verification: There is no principled mechanism to verify that benchmark images strictly adhere to their "negative ground truth" (i.e., that the target is truly absent or illegible by design).

2. Methodology

The authors propose Ghost-100, a procedurally constructed benchmark and a dual-track evaluation framework designed to isolate linguistic tone as the sole independent variable.

A. The Ghost-100 Benchmark

• Scale: 800 synthetically generated images across 8 categories.
• Negative-Ground-Truth Principle: Every image is constructed to guarantee the queried target is absent, illegible, or indeterminate. This eliminates annotation ambiguity found in natural image datasets.
• Task Families:
  1. Text-Illegibility (Categories 01–03): Text is present but blurred, replaced by gibberish, or blank.
  2. Time-Reading (Categories 04–06): Clocks or timers are structurally obscured or logically indeterminate.
  3. Object-Absence (Categories 07–08): Objects are physically missing despite strong contextual cues (e.g., a scene schema implying a missing object, or human intent implying a missing tool).
• 5-Level Prompt Intensity Framework: Each image is paired with five prompts ranging from L1 (Neutral/Cautious) to L5 (Coercive/Explicit Demand). The image and task identity remain fixed; only the directive force of the prompt changes.

B. Dual-Track Evaluation Protocol

To decouple the frequency of hallucination from its severity, the authors introduce two metrics:

1. H-Rate (Rule-Based): A binary metric measuring the proportion of responses where the model crosses from a grounded refusal/abstention into an unsupported positive commitment.
   • Policy: Uses category-specific whitelists to distinguish between hallucinating a target vs. correctly reading non-target background text.
2. H-Score (LLM-as-Judge): A 1–5 severity scale evaluated by GPT-4o-mini (using a "pure-GPT" mode where the judge sees only the prompt and response, not the image).
   • Level 1–2: Refusal or grounded non-target reporting (Non-hallucinated).
   • Level 3: Tentative/Inferred positive commitment (Weak hallucination).
   • Level 4: Confident specific false claim.
   • Level 5: Elaborate fabrication with invented visual details (e.g., color, font, location).

C. Automated Validation Workflow

A three-stage pipeline was developed to ensure dataset integrity:

1. Perceptual Screening: Filters images based on clarity, realism, and missing-element visibility.
2. Design Compliance Verification: Uses GPT-4o-mini to verify that the target is strictly absent/illegible and that no disallowed tokens are visible.
3. Semantic Distinguishability: Uses unsupervised clustering to ensure images from different categories remain visually separable.

• Result: 717 out of 800 images passed strict compliance criteria.

3. Key Contributions

1. Ghost-100 Benchmark: An expanded, procedurally generated dataset (800 images, 8 categories) that isolates tone as a variable under negative-ground-truth conditions.
2. Dual-Track Evaluation Framework: A novel scheme separating H-Rate (occurrence) from H-Score (intensity/confidence), revealing that models can be robust in frequency but fragile in severity, or vice versa.
3. Automated Validation Pipeline: A reusable, three-stage quality assurance tool for verifying negative-ground-truth constraints in synthetic benchmarks.
4. LLM-as-Judge Scoring: A standardized rubric for quantifying the "richness" of hallucinations without relying on human annotators for every sample.

4. Experimental Results

The framework was evaluated on nine open-weight VLMs (including Qwen, InternVL, LLaVA, Llama, Gemma, and DeepSeek families).

• Dissociation of Metrics: H-Rate and H-Score often diverge. Some models have low H-Rates but high H-Scores (rarely hallucinating, but when they do, they lie confidently), while others show the opposite.
• Task-Specific Robustness:
  • Presence-Detection Tasks (Obj-Absence): Several models (e.g., InternVL2.5-8B) showed near-zero hallucination regardless of prompt pressure.
  • Reading-Style Tasks (Text/Time): These subsets were significantly more susceptible to tone-induced hallucination.
• Non-Monotonic Sensitivity: Contrary to the assumption that "stronger pressure = more hallucination," several models exhibited non-monotonic behavior. For example, Gemma-4-E4B-IT showed a peak in hallucination at intermediate tone levels (L3) but dropped to near-zero at the highest pressure (L5), suggesting complex interactions between safety alignment and instruction compliance.
• Model Rankings:
  • Most Robust: InternVL2.5-8B (Lowest H-Rate and H-Score).
  • Most Vulnerable: DeepSeek-VL-7B (Highest H-Rate and H-Score).
  • Generational Improvement: Qwen3-VL-8B was more robust than Qwen2.5-VL-7B.

5. Significance and Implications

• Linguistic Pressure as a Failure Mode: The study proves that hallucination is not just a static perceptual error but a dynamic behavior modulated by linguistic framing. Safety-aligned models can be bypassed by structured, escalating coercion without triggering standard refusal mechanisms.
• Beyond Binary Metrics: The findings argue that aggregate metrics (e.g., "accuracy") obscure critical failure modes. A model might appear robust because it rarely hallucinates, but when it does, it may provide dangerously specific false information (High H-Score).
• Safety Alignment Limitations: The non-monotonic results suggest that current alignment procedures (RLHF) are sensitive to semantic hostility but may fail under structural prompt pressure, leading to unpredictable "spikes" in fabrication at intermediate pressure levels.
• Methodological Standard: The release of the automated validation workflow provides a necessary tool for future research to ensure benchmark integrity, moving away from error-prone manual auditing of synthetic datasets.

In conclusion, Ghost-100 and its evaluation framework provide a granular instrument for studying how VLMs negotiate conflicting objectives (helpfulness vs. truthfulness) under pressure, highlighting the need for tone-robust fine-tuning and uncertainty-aware decoding strategies.