Multimodal LLMs Do Not Compose Skills Optimally Across Modalities

This paper reveals that Multimodal Large Language Models (MLLMs) struggle to optimally compose skills across different modalities, exhibiting significant performance gaps even when employing strategies like chain-of-thought prompting and specialized fine-tuning.

The Gist

Imagine you have a brilliant assistant who is incredibly smart at two very different things:

1. Reading Text: They can solve complex math problems and answer trivia questions just by reading.
2. Looking at Pictures: They can identify objects, read signs in photos, and count items in a scene.

You would think that if you asked this assistant to look at a picture of a math problem and then solve it, they would be perfect. After all, they are great at reading, and they are great at math.

The paper's big discovery is that they aren't.

Even though the assistant is a master of both skills individually, when you ask them to combine them to solve a single problem, they stumble. It's like a world-class chef who can bake a perfect cake and can also grill a perfect steak, but if you ask them to make a "Steak-and-Cake Sandwich," they get confused and serve you a burnt mess.

The Three "Test Drives"

The researchers put several popular AI models (like LLaVA, Qwen, and Molmo) through three simple tests to see how well they could "stitch" these skills together:

1. The "Photo of a Math Test" Challenge:

   • The Task: Show the AI a photo of a math question written on paper (e.g., "What is 7 + 5?").
   • The Skills Needed: First, the AI must read the text in the photo (like a scanner). Second, it must do the math.
   • The Result: The AI often misreads the numbers in the photo or gets the math wrong, even though it can do both perfectly if given the text directly.

2. The "Count the Oranges" Challenge:

   • The Task: Show the AI a picture of a fruit bowl and ask, "How many oranges are there?"
   • The Skills Needed: First, spot the oranges in the picture. Second, count them using numbers.
   • The Result: The AI might see the oranges but forget to count them, or count them but fail to identify them as oranges.

3. The "Card Game" Challenge:

   • The Task: Show the AI a hand of playing cards and ask it to sort them or calculate a score based on complex rules (e.g., "Sum the value of all red cards").
   • The Skills Needed: First, identify the cards (Is that a 7 of Hearts?). Second, apply the rules (Hearts are red, so double the value).
   • The Result: The AI often gets the card identity right but messes up the math, or vice versa.

The "Cascaded" Hack

To prove the AI was capable of these skills, the researchers tried a "cheat code" called Cascaded Inference.

Instead of asking the AI to do everything in one go, they forced it to do it in two separate steps:

1. Step 1: "Hey AI, just look at this picture and tell me what text is written there." (The AI does this perfectly).
2. Step 2: "Okay, now take that text you just wrote down and solve the math problem." (The AI also does this perfectly).

The Gap: When they compared the "One-Step" attempt (Direct) to the "Two-Step" cheat code (Cascaded), the Two-Step method was almost always much better. This "gap" proves that the AI isn't failing because it can't read or can't count; it's failing because it can't coordinate the two skills smoothly in one brain.

Why Does This Happen?

Think of the AI's brain like a busy kitchen.

• The Visual Team (eyes) sees the image.
• The Text Team (brain) knows the rules.
• In a "Direct" attempt, the Visual Team shouts the raw data to the Text Team, but the Text Team gets confused by the noise or misinterprets the visual signal.
• In the "Cascaded" approach, the Visual Team writes a clean, organized note, and then hands it to the Text Team. The Text Team is much happier and performs better.

Can We Fix It?

The researchers tried two ways to fix this "coordination problem":

1. The "Step-by-Step" Prompt (Chain-of-Thought):
   They told the AI, "First, look at the image and write down what you see. Then, use that to solve the problem."

   • Result: It helped! The AI got better at organizing its thoughts. But it wasn't a magic fix; it still wasn't as good as the two-step cheat code, and it required a very specific instruction for every new type of problem.

2. Special Training (Fine-Tuning):
   They gave the AI extra homework specifically designed to practice combining these skills.

   • Result: This helped the AI get better at the specific homework it practiced. However, it didn't fully solve the problem for new types of tasks. The AI was still struggling to be a generalist at combining skills.

The Bottom Line

Multimodal AI models are like super-talented specialists who are surprisingly bad at teamwork. They can read and they can see, but they haven't quite learned how to blend those abilities seamlessly.

The paper concludes that we need to figure out why this happens and invent new ways to train these models so they don't just have "eyes" and a "brain," but a single, unified mind that can use both at the same time without tripping over its own feet. Until then, even the smartest AI models will struggle with tasks that seem trivial to a human.

Technical Summary

Here is a detailed technical summary of the paper "Multimodal LLMs Do Not Compose Skills Optimally Across Modalities".

1. Problem Statement

The paper investigates a critical limitation in Multimodal Large Language Models (MLLMs): Cross-Modality Skill Composition. While MLLMs are trained to acquire specific visual skills (e.g., OCR, object detection) and textual skills (e.g., reasoning, counting) separately, it remains unclear whether they can effectively combine these distinct capabilities to solve new, composite tasks.

The authors hypothesize that MLLMs may fail to optimally integrate visual and textual skills when solving tasks that require a sequential combination of both, even when the individual skills are well-learned and the combination is trivial for humans. They define this performance deficit as the "cross-modality skill composition gap."

2. Methodology

A. Evaluation Tasks

The authors designed three specific tasks where a solution requires the sequential composition of one visual skill and one textual skill. These tasks are trivial for humans but test the model's ability to chain capabilities:

1. Reasoning over Rendered Text (Task 1): The model must perform Optical Character Recognition (OCR) on an image containing a text-based math or logic problem, followed by textual reasoning to solve it.
   • Skills: OCR (Visual) + Reasoning (Textual).
2. Object Counting (Task 2): The model must perform object detection on an image to identify specific items, followed by textual counting of the detected instances.
   • Skills: Object Detection (Visual) + Counting (Textual).
3. Card Playing (Task 3): The model must perform image recognition to identify card ranks/suits from an image, followed by numerical reasoning to sort them or calculate a sum based on specific rules.
   • Skills: Image Recognition (Visual) + Numerical Reasoning (Textual).

B. Inference Settings

To measure the composition gap, the authors compared two inference approaches:

1. Direct Inference: The standard approach where the MLLM receives the image and the question in a single prompt and attempts to solve the task end-to-end.
2. Cascaded Inference: A two-step process that manually enforces skill composition.
   • Step 1: The model is prompted to extract the visual information (e.g., OCR text or object bounding boxes).
   • Step 2: The extracted text is fed back into the model with a prompt to perform the textual reasoning task.
   • Significance: This setup isolates the model's ability to perform individual skills. If Cascaded Inference significantly outperforms Direct Inference, it proves the model fails to compose the skills internally.

C. Models and Datasets

• Models: Evaluated popular open-source MLLMs including LLaVA (1.6), Llama 3.2, Molmo, and Qwen2.5-VL (7B and 72B variants). Closed-source models were excluded to ensure transparency regarding training data and skill acquisition.
• Datasets:
  • Task 1: GSM8K, MATH, ARC, MMLU (rendered as images).
  • Task 2: CV-Bench and a new dataset, COCO-Count (created to provide ground-truth bounding boxes for counting analysis).
  • Task 3: Custom datasets based on poker cards (Sort and Sum games).

3. Key Contributions

1. Quantification of the Skill Composition Gap: The paper provides empirical evidence that a significant performance gap exists between Direct and Cascaded inference across almost all evaluated models and tasks. This demonstrates that MLLMs do not naturally compose skills optimally, even for straightforward combinations.
2. Granular Analysis of Failure Modes: By using the Cascaded setup and an "Oracle" setup (where gold-standard visual inputs are provided), the authors distinguish between:
   • Skill Deficiency: The model cannot perform the individual visual or textual skill.
   • Composition Deficiency: The model can perform individual skills but fails to chain them effectively in a single pass.
3. Mitigation Strategies: The authors explored two methods to reduce the gap:
   • Composition-specific Chain-of-Thought (CoT): Explicitly prompting the model to decompose the task into visual and textual steps within a single prompt.
   • Fine-tuning: Training the model on a dataset where visual inputs are paired with solutions generated by the text-only version of the model (forcing the model to learn the mapping from visual input to the reasoning process).

4. Key Results

• Universal Gap: All tested MLLMs exhibited a significant skill composition gap. For example, in Task 1 (Reasoning over text), LLaVA 1.6-Vicuna-13B showed a gap of 22.19 percentage points (Direct: 18.93% vs. Cascaded: 41.12%).
• Model Variance:
  • Qwen2.5-VL-72B showed the smallest gaps (e.g., 0.39 points in Task 1), likely due to its training data including invoices and forms which implicitly teach skill composition.
  • Molmo models showed no gap in Task 2 (Object Counting), but further analysis revealed this was because they performed poorly on both individual skills (detection and counting), rather than successfully composing them.
• Mitigation Effectiveness:
  • CoT Prompting: Improved performance over Direct Inference for most models (e.g., LLaVA-Vicuna improved by ~36 points on ARC-Easy) but generally failed to reach the performance levels of Cascaded Inference. It also lacked scalability as it required task-specific prompt engineering.
  • Fine-tuning: Improved performance and showed some generalization to other datasets. However, it only fully closed the gap when the fine-tuning and evaluation were performed on the exact same dataset distribution.
• Error Sources: In Task 1, the "Oracle" setup (perfect OCR input) still outperformed the Cascaded setup, indicating that OCR errors are a major source of performance degradation, compounding the composition issue.

5. Significance and Conclusion

The paper challenges the assumption that MLLMs are seamlessly integrating multimodal capabilities. The findings suggest that:

• Training Recipes Matter: Models trained on data that implicitly requires skill composition (e.g., Qwen2.5-VL on receipts) perform better than those trained on isolated tasks.
• Current Architectures are Suboptimal: The "late fusion" paradigm (connecting a visual encoder to an LLM) does not guarantee optimal cross-modal reasoning. The models tend to rely on heuristic shortcuts rather than explicitly chaining skills.
• Future Directions: The authors conclude that simply scaling models or adding more data is insufficient. Future research must focus on:
  • Developing training objectives that explicitly reward skill composition.
  • Investigating the internal mechanisms of how MLLMs fail to route information between modalities.
  • Creating more robust evaluation benchmarks that go beyond simple task completion to test compositional generalization.

In summary, the paper establishes that Multimodal LLMs currently possess a "composition bottleneck," where they struggle to combine known visual and textual skills to solve new problems, necessitating new training paradigms to achieve true multimodal reasoning.