LaMI: Augmenting Large Language Models via Late Multi-Image Fusion

LaMI proposes a late multi-image fusion method that augments text-only Large Language Models with test-time visual signals generated via parallel sampling, significantly improving visual commonsense reasoning and NLP performance without the high costs of full multimodal training or the degradation of textual capabilities.

The Gist

Imagine you have a brilliant, world-class librarian (a Large Language Model, or LLM) who has read every book in existence. This librarian knows everything about history, science, and grammar. However, there's one problem: they have never seen the physical world.

If you ask this librarian, "What color is an emperor penguin's belly?", they might guess "yellow" because they've read that penguins are often associated with yellow beaks, or they might just hallucinate an answer based on text patterns. They lack visual grounding.

On the other hand, you have a Vision-Language Model (VLM). This is like a librarian who also has eyes. They can see pictures and answer visual questions perfectly. But to build this "seeing" librarian, you have to retrain them from scratch with millions of image-text pairs. It's expensive, slow, and sometimes, in the process of learning to see, they forget how to be great at pure text tasks.

Enter LaMI (Late Multi-Image Fusion).

The authors of this paper propose a clever, low-cost way to give the "blind" librarian a pair of glasses without rebuilding their entire brain. Here is how it works, broken down into simple analogies:

1. The "Imagination" Phase (Generating Images)

Since the librarian doesn't have a camera, the system asks them to imagine what the object looks like.

• The Analogy: You ask the librarian, "Describe a penguin." Instead of just giving a text answer, the system uses a "dream machine" (a text-to-image generator) to instantly create six different pictures of penguins based on that description.
• Why multiple? One picture might be weird (maybe a penguin in a tuxedo). But if you generate six, you get a variety of perspectives. It's like asking six different artists to draw a penguin; even if one is wrong, the majority will get the belly color right.

2. The "Late Fusion" Phase (The Smart Mixer)

This is the paper's secret sauce. Most previous methods tried to mix the text and the image early in the thinking process, which can confuse the librarian.

• The Analogy: Imagine the librarian is writing a final report.
  • Old Way: They try to look at the picture while they are forming every single sentence. This distracts them and makes their writing messy.
  • LaMI Way: The librarian writes their answer based purely on their text knowledge first. Then, right before they hit "submit," a specialized assistant (the Late Fusion Layer) looks at the six generated pictures and the librarian's draft.
  • The assistant says: "Hey, the librarian wrote 'yellow', but 5 out of 6 pictures clearly show a white belly. Let's correct that to 'white' before we send it."

3. The "Trust but Verify" Mechanism

What if the librarian's imagination is wrong? What if the "dream machine" draws a penguin that looks like a chicken?

• The Analogy: The system uses a "trust meter" (based on CLIP scores). It checks: "Does the generated picture actually match the question?"
  • If the picture is a good match, the system trusts the visual evidence and updates the answer.
  • If the picture is garbage or irrelevant, the system ignores the image and sticks with the librarian's original text answer. It's like a safety net that only catches you if you're actually falling.

Why is this a big deal?

1. It's Cheap: You don't need to retrain the giant AI model. You just add a small "adapter" and a few seconds of image generation time.
2. It's Flexible: You can use this on any powerful text model (like LLaMA 3) instantly.
3. It Doesn't Break Things: Unlike other methods that make the AI worse at text tasks when they try to add vision, LaMI actually makes the text tasks better or keeps them the same, while fixing the visual blind spots.

The Bottom Line

LaMI is like giving a blind genius a set of instant, on-demand sketches to help them solve a puzzle. Instead of forcing the genius to learn how to see from scratch, you just show them a few quick drawings right before they give their final answer. If the drawings make sense, they use them; if not, they ignore them. The result? A smarter, more grounded AI that understands the world, not just words about it.

Technical Summary

1. Problem Statement

Large Language Models (LLMs) trained exclusively on text excel at textual reasoning but lack visual grounding, leading to failures in visual commonsense tasks (e.g., knowing the color of an emperor penguin's belly). While Vision-Language Models (VLMs) address this, they suffer from two main drawbacks:

1. Performance Degradation: They often perform worse than text-only LLMs on purely textual commonsense reasoning.
2. High Adaptation Cost: Integrating new, powerful LLMs into VLMs requires expensive, full-scale multimodal retraining.

Existing "Visually-Augmented LLMs" (VaLMs) attempt to inject visual signals into pretrained LLMs without full retraining. However, prior approaches typically rely on early fusion (injecting visual tokens deep into the model stack) and single-image inputs. These methods can disrupt the LLM's language capabilities, introduce noise, and fail to capture diverse visual evidence.

2. Methodology: LaMI (Late Multi-Image Fusion)

The authors propose LaMI, a framework that enhances LLMs with visual cues at inference time without compromising their original text-only performance. The method consists of two core innovations:

A. Late Fusion Architecture

Instead of feeding visual tokens into the LLM's intermediate layers (early fusion), LaMI employs a Late Fusion Attention Layer (LFAL):

• Components: A frozen pre-trained LLM, a frozen vision encoder (CLIP), a trainable Visual Token Projector (VTP), and the trainable LFAL.
• Mechanism:
  1. The vision encoder extracts features from an image, which the VTP projects into "pseudo-text embeddings."
  2. The LLM processes the input text independently to produce token embeddings.
  3. Late Integration: Just before the final prediction head, a single attention mechanism allows the text tokens to attend to the projected visual tokens.
• Benefit: This keeps the LLM's internal language reasoning intact while allowing it to access visual information only when necessary for the final decision.

B. Multi-Image Generation & Aggregation

Since paired images are unavailable at inference, LaMI generates visual evidence dynamically:

1. Generation: For a given text prompt, the system generates $k$ diverse images using a distilled text-to-image model (SDXL-turbo) via batched, parallel sampling.
2. Processing: Each generated image is processed through the late-fusion module to produce a probability distribution. A text-only distribution is also computed.
3. Entropy-Aware Weighting: The final prediction is a weighted sum of the $k$ image-based distributions and the text-only distribution.
   • The weight is determined by a CLIP alignment score between the generated image and the input text.
   • High-alignment images (confident visual evidence) are trusted more.
   • Low-alignment images (irrelevant or hallucinated) cause the model to fall back to the text-only prediction.

3. Key Contributions

1. Novel Architecture: Introduction of a Late Fusion mechanism that integrates visual features only at the final prediction stage, preserving the LLM's native language capabilities.
2. Multi-Image Evidence: A strategy to generate and aggregate multiple diverse images at inference time, mitigating the noise and bias inherent in single-image approaches.
3. Efficient Adaptation: The method requires training only lightweight components (projector and fusion layer) on top of frozen foundation models, enabling rapid adaptation to new LLMs (e.g., LLaMA 3) without full multimodal retraining.
4. Dual Improvement: Unlike VLMs that trade text performance for vision, LaMI improves visual commonsense while maintaining or even improving text-only reasoning performance.

4. Experimental Results

The authors evaluated LaMI on object commonsense, visual commonsense, and standard NLP benchmarks across models ranging from GPT-2 to LLaMA 3.

• Object Commonsense: LaMI significantly outperforms prior VaLMs (e.g., VaLM, Z-LaVI, LiVE) and text-only baselines.
  • Example: On "Memory Color," LaMI achieved 74.5% vs. 54.0% for the previous best VaLM.
• Visual Commonsense & Reasoning: LaMI matches or exceeds the performance of heavy VLMs (like InstructBLIP and LLaVA-Next) on vision-heavy tasks (ImageNetVC, PIQA) while avoiding the performance drop on text tasks (Reading Comprehension, Commonsense Reasoning) often seen in VLMs.
• Scaling: When applied to strong models like LLaMA 3-8B, LaMI improved both visual commonsense and text-only performance.
• Ablation Studies:
  • Late vs. Early Fusion: Late fusion consistently outperformed early and intermediate fusion, especially on shape and size reasoning.
  • Multi-Image vs. Single Image: Generating multiple images ($k \approx 6$) provided significant gains over single-image inputs.
  • Generation vs. Retrieval: Generating images from text proved superior to retrieving existing images, offering better specificity and diversity.
  • Compute Baseline: Comparing LaMI against a "Best-of-N" text-only sampling strategy (with matched compute budget) showed that LaMI's gains come from grounded visual evidence, not just increased compute.

5. Significance and Limitations

Significance:

• Test-Time Scaling: LaMI demonstrates that scaling test-time compute (via image generation) is a principled way to enhance reasoning, aligning with recent trends in "agentic" AI.
• Practicality: It offers a low-cost, plug-and-play solution to give text-only LLMs visual grounding without the massive resource requirements of training new VLMs.
• Robustness: The entropy-aware weighting ensures the model does not hallucinate when visual evidence is weak, falling back to reliable text priors.

Limitations:

• Inference Latency: Generating images adds computational overhead (approx. 50ms per image), making it slower than pure text decoding.
• Hallucination Risks: If the text-to-image generator creates misleading images (e.g., for abstract concepts like the "Sword of Damocles"), the high alignment score might incorrectly override the correct text prior.
• Training Data: The method relies on existing pre-trained models (LLM, CLIP, Diffusion), inheriting their biases and potential factual inaccuracies.

In conclusion, LaMI represents a significant step forward in efficiently bridging the gap between text-only LLMs and visual reasoning, proving that late fusion of multi-generated visual evidence is a robust strategy for enhancing commonsense without sacrificing linguistic capability.