💻 computer science

YARD: Y-Architecture Register Decoding for Efficient Hallucination Mitigation in Large Vision-Language Models

YARD is a training-free framework that mitigates hallucinations in Large Vision-Language Models by employing a Y-architecture to share shallow computations and branch at middle layers, where it replaces fine-grained visual tokens with register tokens to generate an effective contrastive signal without the latency of a second forward pass.

Original authors: Ting Chen, Geng Li, Guohao Chen, Yu Hu, Guan Huang, Mai Chen, Langsheng Lei, Jun Du

Published 2026-06-01

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Ting Chen, Geng Li, Guohao Chen, Yu Hu, Guan Huang, Mai Chen, Langsheng Lei, Jun Du

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine a Large Vision-Language Model (LVLM) as a very talented but slightly overconfident storyteller. You show it a picture of a beach, and it starts describing the scene. Sometimes, it gets things right ("There are umbrellas and people"), but other times, it confidently invents details that aren't there ("And look, there's a surfboard and a dog!"). This is called hallucination.

The paper introduces a new method called YARD (Y-Architecture Register Decoding) to stop this storyteller from making things up, without needing to retrain the model or slow it down.

Here is how YARD works, broken down into simple concepts:

1. The Problem with Old Methods

Before YARD, researchers tried to fix hallucinations using a technique called "Contrastive Decoding." Think of this as asking the storyteller to tell the story twice:

Version A (Clean): "Look at the real photo and tell me what you see."
Version B (Degraded): "Tell me what you see, but pretend the photo is blurry or missing pieces."

The computer then compares the two stories. If Version B invents a surfboard but Version A doesn't, the computer knows to suppress the "surfboard" idea in the final answer.

However, previous methods had two big flaws:

The "Blindfold" Approach: Some methods completely removed the visual input for Version B. This was too extreme. The model would just guess based on general knowledge (e.g., "Beaches usually have surfboards"), which didn't help catch specific lies about this specific beach.
The "Double-Work" Approach: Other methods corrupted the image pixels (adding noise). This required the computer to process the image twice from scratch, which was very slow and expensive.

2. The YARD Solution: The "Y" Shape

YARD changes the game by building a Y-shaped path inside the model's brain.

The Stem (Shared Path): Both the "Clean" story and the "Degraded" story start together. They share the first few layers of processing. This is like reading the first paragraph of a book together. This saves time because the model doesn't have to do double the work.
The Split (The Middle Layer): At a specific point in the middle of the model's thinking process, the path splits.
- The Left Branch (Clean): Continues normally, looking at every tiny detail (patches) of the image.
- The Right Branch (Degraded): This is where the magic happens. Instead of looking at every tiny detail, this branch is forced to look at the image through a "Register."

3. The "Register" Metaphor

Imagine the image is a high-resolution map.

The Clean Branch looks at the map and sees every street, tree, and house.
The Degraded Branch (using YARD) is given a "Register." A Register is like a summary note that says, "This is a beach scene with water and sand," but it hides the specific details. It knows there are people, but it can't see where they are standing or what they are holding.

Why is this clever?
If the model tries to say, "There is a surfboard," the Clean Branch (which sees the details) checks the map and says, "No, I don't see a surfboard." The Degraded Branch (which only sees the general beach vibe) might say, "Well, beaches often have surfboards, so maybe?"

When the computer compares these two, it realizes: "The Clean Branch says 'No,' but the Degraded Branch is guessing 'Yes' based on general vibes." YARD then suppresses the "surfboard" guess. It keeps the story grounded in reality because the "Clean" branch has the proof, and the "Degraded" branch acts as a detector for things the model is just guessing.

4. Why the "Middle" Matters

The paper discovered that the model's brain works in stages.

Early layers: Just getting the picture ready.
Middle layers: This is the "sweet spot" where the model starts connecting the text to the specific visual details.
Late layers: The model has already decided what to say.

YARD splits the path right in the middle. If you split too early, the model hasn't seen enough of the picture yet. If you split too late, the model has already "memorized" the details and won't be fooled by the degraded version. The middle is the perfect time to introduce the "Register" to test if the model is actually looking at the picture or just guessing.

5. The Result

By using this Y-shaped path and the "Register" trick:

It's Faster: Because the two paths share the beginning, the model doesn't have to process the image twice from scratch.
It's Smarter: It catches hallucinations better than previous methods because it creates a "fair fight" between a detailed view and a general view, rather than a detailed view vs. a blind guess.
It Works Everywhere: The paper tested this on many different types of AI models, and it worked consistently well without needing to retrain them.

In short, YARD is like a fact-checker that sits inside the AI's brain, asking, "Are you sure you see that, or are you just guessing based on what usually happens?" It does this efficiently, without slowing down the conversation.

Technical Summary: YARD – Y-Architecture Register Decoding for Efficient Hallucination Mitigation

1. Problem Statement

Large Vision-Language Models (LVLMs) frequently suffer from hallucinations, generating text that is semantically plausible but inconsistent with visual input (e.g., non-existent objects, incorrect attributes, or unsupported spatial relations). While Contrastive Decoding (CD) offers a training-free solution by contrasting a "clean" model branch with a "degraded" branch to suppress hallucination-prone logits, existing methods face two critical dilemmas:

Sub-optimal Degraded Branches:
- Text-only approaches: Completely dropping visual tokens reduces the degraded branch to a language-prior model. This induces hallucinations based on generic priors that may not align with the specific input image, failing to isolate image-specific hallucinations.
- Pixel-level degradation: Corrupting input images (e.g., adding noise) offers coarse control. Because visual features (color, texture, boundaries) are tightly entangled in pixel space, these perturbations often fail to isolate local evidence responsible for hallucinations and instead contaminate the degraded logits with genuine visual signals.
Inference Latency: Most existing CD methods require two full forward passes (one for the clean branch, one for the degraded branch), significantly increasing inference latency.

2. Methodology: YARD

The authors propose YARD (Y-Architecture Register Decoding), a training-free framework that constructs a degraded branch internally within the LLM decoder. The method is driven by two key observations regarding cross-modal information flow:

A. Where to Degrade: The Middle Layer Window

Analysis of text-to-vision attention and intervention experiments reveals that:

Shallow layers: Attention is diffuse and easily distracted.
Deep layers: Task-relevant visual evidence has largely been integrated into text-side representations; perturbing vision tokens here has minimal impact on final predictions.
Middle layers: This is the critical window where reliable visual grounding emerges, and text tokens begin to acquire local visual evidence before full integration.
YARD Strategy: The framework shares the shallow decoder layers between the clean and degraded branches and splits (branches) exactly at a middle layer (e.g., layer 10 in a 25-layer model). This allows the degraded branch to inherit early multimodal context while preventing the full integration of fine-grained local evidence.

B. How to Degrade: Register-Based Feature-Level Degradation

Instead of corrupting the input image or removing tokens entirely, YARD constructs a degraded condition at the feature level:

Register Tokens: YARD utilizes register tokens (originally proposed to absorb high-norm outlier artifacts in Vision Transformers). These tokens are constructed at test time by redirecting anomalous MLP activations to an appended token.
The Degraded Mechanism:
- Clean Branch: Retains original patch-level visual tokens ( $Z$ ) and register tokens ( $R$ ).
- Degraded Branch: Discards the fine-grained patch tokens ( $Z$ ) and relies only on register tokens ( $R$ ) and text tokens ( $T$ ).
Effect: Register tokens preserve coarse global image semantics (keeping the branch "image-aware") but lack fine-grained local correspondence (making it "locally under-grounded"). This creates a contrastive signal that exposes hallucination-prone components without the extreme modality mismatch of text-only branches or the noise of pixel-level corruption.

C. Y-Architecture for Efficiency

By sharing the computation of the shallow layers ($1$ to $K$ ) before the split, YARD avoids the costly second full forward pass required by input-level contrastive methods. The branches only diverge for the remaining layers ( $K+1$ to $D$ ).

The final logit is computed as:
$\ell^{yard}_i = (1 + \alpha)\ell^c_i - \alpha\ell^d_i$
where $\ell^c$ and $\ell^d$ are the logits from the clean and register-conditioned degraded branches, respectively.

3. Key Contributions

Register-Based Feature-Level Degradation: Introduces a novel method to construct an image-aware but locally under-grounded contrastive branch without corrupting the visual input, utilizing register tokens to preserve global semantics while removing local evidence.
Analysis of Degraded-Branch Purity: Demonstrates through ablation that feature-level degradation (specifically Global Info. via registers) exposes a purer hallucination signal than text-only or pixel-level degradation, as evidenced by lower hallucination metrics when the conservative clean-logit anchoring ( $\alpha$ ) is removed.
YARD Framework: Presents an efficient, training-free Y-Architecture decoding framework that consistently mitigates hallucinations across multiple LVLMs while significantly reducing inference latency compared to dual-pass methods.

4. Experimental Results

YARD was evaluated on both generative (AMBER, Object HalBench, MME-Hallucination) and discriminative (AMBER Discrimination, POPE) benchmarks using diverse backbones (LLaVA-1.5, LLaVA-NeXT, Qwen2/3-VL, InstructBLIP, Mini-Gemini).

Generative Performance: On LLaVA-1.5-7B, YARD reduced Object HalBench scores (CHs/CHi) from 27.0/10.5 to 19.7/6.5, outperforming state-of-the-art methods like OPERA, VCD, and FuzzyCD. It also achieved the best color perception scores on MME-Hallucination, indicating preserved visual capability.
Discriminative Performance: YARD improved AMBER accuracy/F1 from 67.00/71.10 to 78.70/84.10 and POPE accuracy/F1 from 83.93/82.24 to 86.14/85.85 on LLaVA-1.5.
Transferability: The method demonstrated consistent improvements across all tested architectures, including Qwen3-VL and LLaVA-NeXT, suggesting it captures a general property of LVLM decoding rather than being architecture-specific.
Ablation Studies: Confirmed that branching at the middle layer (e.g., $K=10$ ) is critical; branching too early or too late yields suboptimal results. Register-based degradation outperformed "Average" feature degradation and "Text-Only" variants.

5. Significance and Claims

The paper claims that YARD addresses the fundamental trade-off in contrastive decoding: the need for a degraded branch that is sufficiently different to expose hallucinations but sufficiently aligned with the image to avoid generic language priors. By moving degradation into a structured feature-level space within the decoder, YARD achieves:

Faithful Contrastive Signals: The "global-but-local-missing" design effectively suppresses predictions that are plausible under weak local grounding but unsupported by clean visual evidence.
Efficiency: The Y-architecture eliminates the need for a complete second forward pass, making it a practical solution for deployment.
Generality: The approach is not tied to specific model architectures, offering a general training-free heuristic for hallucination mitigation.

The authors note limitations regarding the reliance on constructing effective register representations for specific Transformer-based LVLMs and the sensitivity to the choice of branching layer, suggesting future work in adapting these mechanisms to broader multimodal architectures.