🤖 AI

Back to the Barn with LLAMAs: Evolving Pretrained LLM Backbones in Finetuning Vision Language Models

This paper systematically investigates the impact of evolving LLM backbones (LLaMA-1 to LLaMA-3) on Vision-Language Models by controlling for other variables, revealing that newer backbones do not universally improve performance but instead alter task-specific behaviors, such as solving different visual questions with better-calibrated confidence, while offering limited benefits for tasks relying primarily on visual understanding.

Original authors: Sameera Horawalavithana, Lauren Phillips, Ian Stewart, Sai Munikoti, Karl Pazdernik

Published 2026-04-14

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Sameera Horawalavithana, Lauren Phillips, Ian Stewart, Sai Munikoti, Karl Pazdernik

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a very smart, well-read Brain (the Large Language Model, or LLM) and a pair of Eyes (the Vision Encoder). To build a robot that can look at a picture and tell you a story about it, you need to connect the Eyes to the Brain.

This paper is like a scientific experiment where the researchers kept the Eyes exactly the same, but they swapped out the Brain three times. They used an older, slightly less advanced brain (LLaMA-1), a medium one (LLaMA-2), and the newest, most sophisticated one (LLaMA-3).

The big question they asked was: "If we give our robot a smarter brain, does it automatically get better at looking at pictures and answering questions?"

Here is what they found, explained simply:

1. A Smarter Brain Doesn't Always Mean a Smarter Robot

You might think, "Newer brain = Better robot." But the researchers found that it depends entirely on the job.

The "Memory Test" Job (ScienceQA): Imagine a test where you have to memorize a textbook chapter and then answer questions based only on that text.
- Result: The newer brains actually did worse here. Why? Because the older brains were really good at memorizing the specific patterns in the training data. The newer, smarter brains started overthinking or relying on their own "common sense" instead of just reading the text, which caused them to make mistakes on this specific type of rigid test.
The "Real World" Job (VQA-Scene): Imagine looking at a photo of a messy living room and answering, "Is there a cat on the sofa?"
- Result: The newer brains did better. They were better at understanding the context and the visual clues, leading to more correct answers.

The Lesson: Upgrading the brain doesn't fix everything. If the job is just about memorizing facts, a newer brain might overcomplicate things. If the job requires real understanding, a newer brain shines.

2. They Are Solving Different Puzzles

The researchers noticed something weird. When they looked at which specific questions the robots got right, they realized the newer robots weren't just getting the same questions right plus a few more. They were getting a completely different set of questions right.

Analogy: Imagine two detectives looking at the same crime scene.
- Detective A (Old Brain) is very confident. They say, "I know who did it!" and point to the butler. They are 100% sure, but sometimes they are wrong.
- Detective B (New Brain) is more humble. They say, "Hmm, it could be the butler, or maybe the gardener." They are less sure of themselves, but they are actually right more often.
- The new brain isn't just "better" at the old way of thinking; it's looking at the clues in a totally different way.

3. The "Eyes" Are Still the Bottleneck

For some jobs, like analyzing a complex geological map to find a specific earthquake location, the researchers found that upgrading the brain did almost nothing.

Analogy: Imagine you have a super-genius brain, but you are trying to read a book written in a language you don't know, or the text is so blurry you can't see the letters. No matter how smart your brain is, if your Eyes (the vision encoder) can't see the details, the brain can't help.
In these cases, the researchers found that the newer brains were just as stuck as the older ones. To fix this, you don't need a smarter brain; you need better glasses (a better vision encoder).

4. The New Brain Can Do Magic the Old One Couldn't

There was one specific task where the newest brain (LLaMA-3) did something the older ones completely failed at: Predicting exact GPS coordinates from a picture of a landscape.

The older brains would look at the map and say, "It's in California."
The new brain looked at the map and said, "It's at Latitude 34.05, Longitude -118.25."
Why? The new brain had been trained on more data that included numbers and maps, so it learned a new "skill" that the older brains simply didn't have. It wasn't just an improvement; it was a brand-new ability.

5. The New Brain is More "Honest"

Finally, the researchers looked at how confident the robots felt about their answers.

The Old Brains were like overconfident students who always raised their hands and shouted the answer, even when they were guessing. They were 100% sure, even when wrong.
The New Brain was more like a careful scholar. It said, "I think the answer is X, but I'm only 70% sure."
Surprisingly, being less confident actually helped the new brain be more accurate. It didn't force an answer; it weighed the possibilities carefully.

Summary

The paper teaches us that in the world of AI, bigger and newer isn't always "better" in a straight line.

If you need a robot to memorize facts, an older, simpler brain might actually work better.
If you need a robot to understand complex scenes, a newer brain helps.
If the robot can't "see" well, giving it a smarter brain won't help; you need better eyes.
And sometimes, a new brain doesn't just get better at the old tricks; it learns entirely new magic tricks that the old ones couldn't even imagine.

1. Problem Statement

Vision-Language Models (VLMs) rely heavily on pre-trained Large Language Models (LLMs) as their reasoning backbones. As LLMs evolve (e.g., from LLaMA-1 to LLaMA-3), there is an assumption that newer backbones automatically yield superior VLM performance. However, existing literature comparing VLMs suffers from significant confounding variables. Previous studies often compare models that differ not only in their LLM backbone but also in vision encoders, training data, hyperparameters, and implementation details. This makes it impossible to isolate the specific impact of the evolving LLM backbone on multimodal capabilities.

The core research question is: How do systematic improvements in LLMs impact VLM downstream task performance when all other factors are held constant?

2. Methodology

The authors conducted a controlled empirical study to isolate the effects of the LLM backbone.

Experimental Design:
- Variable: The LLM backbone was varied across three generations: LLaMA-1, LLaMA-2, and LLaMA-3.1.
- Constants: The vision encoder (CLIP-ViT-Large-Patch14), training datasets, post-training algorithms (Supervised Fine-Tuning), and codebase (NVIDIA NeMo/NeVA architecture) were kept identical across all experiments.
Training Process:
- Stage 1 (Alignment/Warm-up): The model was trained on the SciCap dataset (image captioning for technical plots) to align the visual encoder with the LLM decoder.
- Stage 2 (Supervised Fine-Tuning - SFT): The models were fine-tuned on three distinct domain-specific datasets to evaluate downstream performance.
Evaluation Datasets:
1. ScienceQA: Focuses on science-related questions requiring reasoning over visual and textual information (high reliance on memorization and structured text patterns).
2. VQA-Scene: Focuses on scene understanding with controlled language priors, requiring grounded visual reasoning.
3. Seismic: A multimodal dataset for geological interpretation, involving both text descriptions of features and numerical predictions of latitude/longitude coordinates.
Metrics: Standard accuracy for multiple-choice tasks, BLEU/ROUGE for text generation, Haversine distance for coordinate prediction, and internal analysis of confidence scores and layer-wise context vectors.

3. Key Contributions & Findings

The study yields five major takeaways that challenge the assumption that "newer is always better":

A. Newer LLMs Do Not Uniformly Improve VLM Performance

Performance gains are task-dependent, not universal.

VQA-Scene: Accuracy improved from 61.8% (LLaMA-1) to 65.6% (LLaMA-3).
ScienceQA: Accuracy decreased from 71.8% (LLaMA-1) to 68.4% (LLaMA-3). The authors suggest newer LLMs may rely too heavily on textual shortcuts or language priors, overriding visual evidence in tasks requiring memorization of structured text.
Seismic (Text): Performance remained mixed with no clear trend.

B. Newer Backbones Solve Different Problems, Not Just More

Instance-level analysis revealed that aggregate accuracy masks behavioral shifts.

VQA-Scene: LLaMA-3 is negatively correlated with LLaMA-1 and LLaMA-2 at the instance level ( $r \approx -0.2$ ). LLaMA-3 gets correct a set of questions that the older models get wrong, and vice versa. It does not simply solve the same questions better; it engages with visual information differently.
ScienceQA: Correlations drop across generations, indicating a shift in which questions are solved.
Seismic: High correlation ( $r > 0.85$ ) across all models, as the task is bottlenecked by the visual encoder.

C. Vision Bottlenecks Limit LLM Upgrades

For tasks where visual perception is the primary constraint (e.g., Seismic text descriptions), upgrading the LLM backbone offers minimal returns. Since the CLIP encoder was held constant, the models could not extract better geological features, rendering LLM improvements irrelevant for these specific sub-tasks.

D. Emergence of New Capabilities

Newer LLM backbones can unlock capabilities that older generations completely lack.

Seismic Coordinate Prediction: LLaMA-1 and LLaMA-2 scored 0.0% on predicting latitude/longitude coordinates (failing to format numbers correctly). LLaMA-3 achieved 76.5% accuracy. This suggests that structured numerical output and domain-specific formatting capabilities emerge only with sufficiently advanced backbones.

E. Internal Mechanisms: Confidence and Representation

The study analyzed the internal dynamics of the models:

Calibration: LLaMA-1 and LLaMA-2 were overconfident (mean confidence $\approx 1.0$ ) regardless of correctness. LLaMA-3 was less confident (mean confidence 0.73) but more accurate, indicating better-calibrated predictions.
Layer-wise Dynamics:
- LLaMA-1/2: Exhibit large activation magnitudes and significant, unstable shifts in context representation in deeper layers (negative cosine similarity between adjacent layers).
- LLaMA-3: Exhibits smaller, more stable activations and incremental refinements across layers (positive, stable similarity). This stable internal pathway correlates with its ability to solve different instances and its better calibration.

4. Significance and Implications

Rigorous Evaluation: The paper establishes a benchmark for evaluating VLMs by controlling for confounding variables, proving that LLM upgrades do not guarantee VLM improvements.
Task-Specific Optimization: Practitioners should not blindly upgrade LLM backbones. For perception-heavy tasks, improving the vision encoder is more critical. For reasoning tasks, the nature of the reasoning (memorization vs. grounded visual) dictates whether an upgrade helps.
Emergent Abilities: Newer generations may introduce entirely new capabilities (e.g., structured numerical output) rather than just incremental improvements.
Internal Analysis: The study highlights that aggregate accuracy is insufficient for understanding model behavior. Differences in internal representation stability and confidence calibration are crucial for understanding why a model succeeds or fails.

In conclusion, the paper argues that the relationship between LLM evolution and VLM performance is non-linear and task-dependent. Newer backbones change the mechanism of reasoning and the types of problems solved, rather than simply providing a uniform performance boost.