Modality Inflation: Energy Characterization and Optimization Opportunities for MLLM Inference

This paper investigates "modality inflation" in multimodal large language models by providing the first stage-level energy analysis across vision encoding, prefill, and decoding, revealing significant energy overheads and GPU underutilization while demonstrating that stage-wise dynamic voltage and frequency scaling (DVFS) offers an effective optimization strategy for energy-efficient inference.

The Gist

Imagine you have a brilliant, super-fast chef (the AI) who is famous for writing amazing recipes based on text instructions. This chef is so good that they can cook a whole meal just by reading a shopping list.

Now, imagine you want to upgrade this chef so they can also look at photos of ingredients before cooking. You add a new assistant, a "Visual Inspector," to the kitchen. This assistant looks at the photos, describes them in detail, and hands the description to the chef.

This paper is about the energy bill of running this upgraded kitchen. The researchers discovered that while adding photos makes the chef smarter, it also makes the kitchen much more expensive to run, and the cost depends entirely on how you set up the kitchen.

Here is the breakdown of their findings using simple analogies:

1. The Problem: "Modality Inflation" (The Balloon Effect)

When you give the chef just a text list, the work is straightforward. But when you add photos, two things happen that "inflate" the workload:

• The Inspector's Job: The Visual Inspector has to look at the photo, analyze it, and turn it into a long list of words (tokens) for the chef.
• The Chef's Burden: The chef now has to read a much longer list (the original text + the photo description) before they can start cooking.

The researchers call this "Modality Inflation." It's like trying to fit a giant, water-filled balloon into a small box. The balloon (the data) gets bigger and heavier, requiring more energy to push it through the pipeline.

2. The Big Surprise: Not All Kitchens Are Equal

The team tested four different kitchen setups (different AI models). They found that adding photos didn't cost the same amount of energy for everyone.

• Model A (The Efficient Chef): Adding photos only increased the energy bill by 17%.
• Model B (The Heavy Lifter): Adding photos skyrocketed the energy bill by 94% (almost double!).

The Lesson: You can't treat all AI models the same. Some are built with a "heavy-duty" inspector that burns a lot of energy just looking at the photo. Others have a "lazy" inspector but then dump a massive amount of data onto the chef, causing a traffic jam later.

3. Where is the Energy Going? (The Three Stages)

The researchers broke the cooking process into three stages to see where the money was being wasted:

• Stage 1: The Visual Inspector (Encoding)
  • Analogy: This is the assistant staring at the photo and writing a report.
  • Finding: For some models, this stage is the energy hog. It's like having an assistant who uses a 100-watt lightbulb just to read a single picture.
• Stage 2: The Prep Work (Prefill)
  • Analogy: This is the chef reading the entire list (text + photo report) before picking up a knife.
  • Finding: If the photo report is too long (because the image was high-resolution or there were many images), the chef spends a huge amount of energy just reading the list. This is where the "traffic jam" happens.
• Stage 3: Cooking (Decoding)
  • Analogy: The actual cooking.
  • Finding: This part is surprisingly stable. Whether the input was text or photos, the energy to cook the final dish is roughly the same. The waste happens before the cooking starts.

4. The "Idle" Problem

The researchers looked at the power meter on the kitchen's main generator (the GPU).

• Text-only: The generator revs up to maximum power immediately and stays there. It's a "race to the finish."
• With Photos: The generator often runs at a "medium hum" for a long time while the Visual Inspector does their job. It's like driving a car in second gear for a long time instead of shifting to a higher gear. The current settings often keep the engine revving too high for this "medium" work, wasting fuel.

5. The Solution: Smart Dimmer Switches (DVFS)

The paper suggests a clever fix called Dynamic Voltage and Frequency Scaling (DVFS).

Think of the AI's processor as a car engine with a dimmer switch for its speed.

• Current approach: We keep the engine at 100% speed (high frequency) all the time, just in case.
• New approach: We use a smart dimmer.
  • When the Visual Inspector is working (which is heavy but doesn't need to be instant), we slow the engine down. It takes a tiny bit longer, but we save a lot of fuel.
  • When the Chef is cooking (which needs to be fast), we speed the engine up.

The Result: By adjusting the speed based on which part of the process is happening, they saved a significant amount of energy without making the AI noticeably slower.

Summary

This paper tells us that making AI "see" images is great, but it's currently very expensive and inefficient because we are using a "one-size-fits-all" approach.

• The Issue: Adding images creates a "balloon" of extra data that wastes energy.
• The Discovery: Different AI models waste energy in different ways (some in the "looking" phase, some in the "reading" phase).
• The Fix: We need to be smarter. We should slow down the AI when it's doing heavy, non-urgent work (like analyzing a photo) and speed it up only when it's doing the urgent work. This is like using a dimmer switch instead of leaving the lights on full blast.

By doing this, we can make the next generation of "seeing" AI much greener and cheaper to run.

Technical Summary

Here is a detailed technical summary of the paper "Modality Inflation: Energy Characterization and Optimization Opportunities for MLLM Inference."

1. Problem Statement

Multimodal Large Language Models (MLLMs) extend text-only LLMs by integrating vision encoders to process images and videos. While this enables broader applications, it introduces a phenomenon the authors term "Modality Inflation."

• The Core Issue: Multimodal inputs require an additional vision encoding stage and generate hundreds to thousands of visual tokens. These tokens are concatenated with text tokens, drastically expanding the input sequence length for the LLM backbone.
• The Gap: Existing research focuses heavily on text-only LLMs or throughput optimization. There is a lack of understanding regarding:
  • The specific energy overhead introduced by multimodal inputs compared to text-only baselines.
  • How energy is distributed across the inference pipeline (encoding, prefill, decoding).
  • How input complexity (resolution, image count) scales energy consumption across different architectures.
  • The potential for stage-specific energy optimization strategies.

2. Methodology

The authors conducted a rigorous, stage-level energy characterization of MLLM inference on an NVIDIA A100 80GB GPU.

• Models Evaluated: Four representative MLLMs with diverse architectures and tokenization strategies:
  • LLaVA-1.5: Uses fixed-patch tokenization (CLIP ViT).
  • LLaVA-OneVision: Uses any-resolution tiling (SigLIP ViT), generating massive token counts.
  • Qwen2.5-VL: Uses a compute-intensive encoder (QwenViT).
  • InternVL3: Uses compressed ViT-based encoding.
  • Control: All models utilize LLM backbones in the 7B–8B parameter range to isolate the impact of the vision encoder and tokenization.
• Experimental Setup:
  • Iso-token Comparison: Compared MLLM inference against text-only baselines where the total input token count (text + visual) was matched to pure text tokens to isolate the cost of the modality itself.
  • Stage Decomposition: The pipeline was broken down into three distinct stages: Vision Encoding, Prefill, and Decoding.
  • Metrics: Measured Energy per Request (Joules), End-to-End Latency (seconds), and GPU Power Traces (Watts) at 5ms intervals using NVML.
  • Variables: Tested sensitivity to Image Resolution (224px to 2048px) and Image Count (1 to 6 images).
• Optimization Evaluation: Investigated Stage-wise Dynamic Voltage and Frequency Scaling (DVFS) to determine if adjusting GPU frequency per stage could reduce energy without violating latency constraints.

3. Key Contributions

1. First Stage-Level Energy Characterization: Provided the first detailed breakdown of energy consumption across encoding, prefill, and decoding stages for MLLMs.
2. Quantification of Modality Inflation: Quantified the energy overhead of multimodal inference, revealing it ranges from 17% to 94% higher than text-only baselines for identical token counts, depending on the architecture.
3. Identification of Architecture-Dependent Bottlenecks: Demonstrated that energy bottlenecks are not uniform; they shift between the compute-heavy vision encoder (e.g., Qwen2.5-VL) and the prefill stage caused by token explosion (e.g., LLaVA-OneVision).
4. Input Complexity Analysis: Mapped how resolution and image count drive non-linear energy scaling, showing that high-resolution inputs can disproportionately increase prefill costs.
5. DVFS Optimization Strategy: Proved that stage-wise DVFS is an effective optimization, allowing significant energy savings by downclocking during specific stages (like encoding) where latency sensitivity is lower.

4. Key Results & Findings

A. Energy Overhead & Architecture Variance

• Overhead Range: Multimodal inference incurs 17% to 94% additional energy compared to text-only baselines.
  • Qwen2.5-VL: Highest overhead (+94%) due to a compute-heavy encoder.
  • LLaVA-OneVision: Lowest relative overhead (+17%) despite generating 3,715 tokens, because its encoder is efficient, though it shifts the burden to the prefill stage.
• Latency vs. Energy: Energy and latency do not scale linearly. For Qwen2.5-VL, a 94% energy increase correlated with a 179% latency increase, indicating low parallelism in the encoder stage.

B. Stage-Level Breakdown

• Vision Encoding: Can be the dominant energy consumer (e.g., Qwen2.5-VL encoder consumed 20.81 J, 6× higher than LLaVA-1.5).
• Prefill: Dominated by token count. LLaVA-OneVision's tiling strategy caused prefill energy to reach 95.78 J (12× higher than a balanced baseline) and latency to increase 8.5×.
• Decoding: Remained stable and largely independent of input modality, driven primarily by output length.

C. Power Profiling & GPU Utilization

• Mid-Power Phases: Unlike text-only inference which quickly saturates the GPU (hitting ~400W), multimodal inference exhibits extended "mid-power phases" (100W–250W) during the vision encoding stage.
• Inefficiency: Default "race-to-idle" policies keep GPUs at high frequencies during these low-utilization phases, wasting energy.

D. Input Complexity Scaling

• Resolution: Models behave differently. LLaVA-1.5 (fixed patch) shows flat energy scaling with resolution. In contrast, Qwen2.5-VL and LLaVA-OneVision show sharp energy spikes at high resolutions (e.g., >1024px) due to uncontrolled token growth.
• Image Count: Energy and latency scale linearly with image count, but the marginal cost per image varies significantly by architecture (15 J/image to 35 J/image).

E. DVFS Optimization Results

• Stage-Wise Tuning: A single global frequency is suboptimal.
  • Encoding Stage: For InternVL3, reducing frequency from 1410 MHz to 1050 MHz increased latency by only 11% but reduced energy by **25%**.
  • Prefill Stage: More sensitive to frequency; downclocking here yields smaller energy gains for larger latency penalties.
• Conclusion: Optimal energy efficiency requires SLO-aware (Service Level Objective) frequency selection that adapts to the specific stage and model architecture.

5. Significance

This paper fundamentally shifts the perspective on MLLM efficiency from a "black box" throughput view to a granular, stage-aware energy view.

• Design Guidance: It proves that "one-size-fits-all" serving policies are inefficient. Systems must be tailored to the specific vision encoder and tokenization strategy of the model.
• Sustainability: By identifying that multimodal workloads spend significant time in underutilized power states, the paper offers a concrete path (stage-wise DVFS) to reduce the carbon footprint of large-scale MLLM deployments.
• Future Systems: The findings advocate for disaggregated serving and dynamic resource allocation where the vision encoder and LLM backbone can be managed independently to maximize energy efficiency.

In summary, the paper establishes that modality inflation is a critical, quantifiable source of energy inefficiency that varies wildly by architecture, and that stage-specific power management is a viable and necessary strategy for sustainable MLLM serving.