💻 computer science

SpikeMLLM: Spike-based Multimodal Large Language Models via Modality-Specific Temporal Scales and Temporal Compression

SpikeMLLM introduces the first spike-based framework for Multimodal Large Language Models that leverages Modality-Specific Temporal Scales and Temporal Compression to achieve near-lossless performance with significantly reduced computational overhead, validated by both benchmark results and a custom RTL accelerator demonstrating superior throughput and power efficiency.

Original authors: Han Xu, Zhiyong Qin, Di Shang, Jiahong Zhang, Xuerui Qiu, Bo Lei, Tiejun Huang, Bo Xu, Guoqi Li

Published 2026-04-22

📖 5 min read🧠 Deep dive

CC BY 4.0

Original authors: Han Xu, Zhiyong Qin, Di Shang, Jiahong Zhang, Xuerui Qiu, Bo Lei, Tiejun Huang, Bo Xu, Guoqi Li

Original paper licensed under CC BY 4.0 (http://creativecommons.org/licenses/by/4.0/). ✨ This is an AI-generated explanation of the paper below. It is not written or endorsed by the authors. For technical accuracy, refer to the original paper. Read full disclaimer

Imagine you have a super-smart robot assistant (a Multimodal Large Language Model, or MLLM) that can read books, look at photos, and understand videos all at once. This robot is incredibly powerful, but it's also a glutton for energy. It eats up massive amounts of electricity and requires huge, expensive computers to run, making it hard to put on a phone or a small robot.

The researchers behind this paper, SpikeMLLM, asked: "How can we make this robot as smart as before, but as energy-efficient as a hummingbird?"

Their answer involves three main ideas: Spiking, Custom Schedules, and Compression.

1. The Robot's New Language: "Spiking"

Most AI models talk in a continuous stream of numbers, like a river flowing constantly. This requires a lot of energy to keep the water moving.

Spiking Neural Networks (SNNs) are different. They talk like fireflies.

The Analogy: Imagine a room full of fireflies. They only flash their lights when they have something important to say. If they have nothing to say, they stay dark and save energy.
The Benefit: Instead of constantly processing data, the robot only "spikes" (fires) when necessary. This is how biological brains work, and it is incredibly energy-efficient.

2. The Problem: One Size Doesn't Fit All

The researchers tried to teach this "firefly robot" to understand both Text (like a novel) and Images (like a photo). They hit two major snags:

Snag A: The Different Paces of Life.
- Text is like a fast-paced conversation. Every word changes the meaning quickly. It needs to be processed very carefully and frequently.
- Images are like a landscape painting. A big part of the picture is just blue sky or green grass (redundant information). It doesn't need to be analyzed as intensely as every single word in a sentence.
- The Mistake: Previous methods tried to treat the text and the image exactly the same way, using the same amount of "time steps" (processing cycles) for both. This wasted energy on the image and didn't give the text enough attention.
Snag B: The Time-Consuming Unfolding.
- To turn a standard computer number into a "firefly flash," the old method had to break the number down into many tiny steps.
- The Analogy: Imagine you want to send a message saying "15". The old way was to tap your finger on the table 15 times, one by one. If you have to send a message saying "1000", you tap 1000 times! This takes forever and uses a lot of time.

3. The Solution: SpikeMLLM's Magic Tricks

To fix these problems, the team invented two clever techniques:

Trick #1: Modality-Specific Temporal Scales (MSTS)

"The Custom Schedule."
Instead of treating the text and image the same, SpikeMLLM gives them different schedules based on how much they change.

The Analogy: Think of a conductor leading an orchestra.
- The Violins (Text) are playing a fast, complex solo. The conductor gives them more time to play their notes clearly.
- The Drums (Images) are just keeping a steady, simple beat. The conductor gives them less time because they don't need to change as much.
The Result: The robot spends its energy where it matters most (the text), saving huge amounts of power on the static parts of the image.

Trick #2: Temporally Compressed LIF (TC-LIF)

"The Binary Code Shortcut."
This fixes the "tapping 1000 times" problem.

The Old Way: To send the number "15", you tap 15 times (1+1+1+1...).
The New Way (TC-LIF): The robot realizes that "15" is actually 1111 in binary code. Instead of tapping 15 times, it taps 4 times, but each tap counts for a different value (1, 2, 4, 8).
- Tap 1 (value 1) + Tap 2 (value 2) + Tap 3 (value 4) + Tap 4 (value 8) = 15.
The Result: Instead of needing 15 time steps, it only needs 4. This is a massive speedup. It's like switching from sending a letter by walking door-to-door to sending it via a high-speed drone.

The Grand Finale: The Hardware

The researchers didn't just stop at the software. They built a specialized chip (an accelerator) designed specifically for this "firefly" style of computing.

The Result: When they tested this new system against a standard, powerful computer chip (GPU):
- It was 9 times faster at processing tokens.
- It used 25 times less power.

Summary

SpikeMLLM is like upgrading a gas-guzzling truck into a high-tech electric vehicle.

It switches the engine to Spiking (only working when needed).
It gives Text the VIP treatment (more time) and Images the economy treatment (less time) because they need different things.
It uses a compression trick to send messages in fewer steps.
It runs on a custom engine built just for this purpose.

The outcome? A super-smart AI that can see and read just as well as the giants of today, but runs on a fraction of the energy, making it possible to put powerful AI into your pocket, your glasses, or even a tiny robot.

1. Problem Statement

Multimodal Large Language Models (MLLMs) have achieved significant breakthroughs but suffer from high computational overhead and energy consumption, hindering deployment in resource-constrained environments. While Spiking Neural Networks (SNNs) offer inherent energy efficiency through sparse, event-driven computation on neuromorphic hardware, extending them to MLLMs presents two critical challenges:

Modality Heterogeneity: Visual and text modalities differ significantly in information density and activation dynamics. A uniform spike encoding strategy fails to accommodate these differences, often leading to insufficient precision for text or redundant computation for images.
Temporal Unfolding Overhead: Existing "integer-to-spike" paradigms map quantized activations to binary spike sequences over $T$ timesteps. For $L$ quantization levels, standard unfolding requires $T = L-1$ timesteps. In MLLMs, high-resolution images generate massive token counts, causing the temporal unfolding overhead ( $T$ ) to become a severe efficiency bottleneck.

2. Methodology: SpikeMLLM

The authors propose SpikeMLLM, the first spike-based framework for MLLMs. It unifies existing ANN quantization methods within the spiking representation space and introduces two core mechanisms to address the challenges above.

A. Modality-Specific Temporal Scales (MSTS)

MSTS addresses the heterogeneity between modalities by adaptively allocating timesteps based on the Modality Evolution Discrepancy (MED).

MED Metric: The authors define MED as $1 - \text{CosineSimilarity}$ between layer $l-1$ and layer $l$ representations. High MED indicates significant representational change, requiring finer temporal resolution.
Strategy: Text tokens typically exhibit higher MED (dynamic evolution) compared to visual tokens (which often have spatial redundancy). MSTS assigns more timesteps to text ( $T_t$ ) and fewer timesteps to visual ( $T_v$ ) modalities.
Layer-wise Allocation: Within each modality, layers with higher MED are prioritized for higher timestep budgets, ensuring representational fidelity where it matters most without increasing the total effective timestep cost.

B. Temporally Compressed LIF (TC-LIF)

TC-LIF addresses the linear growth of timesteps ( $T = L-1$ ) in standard integer-to-spike unfolding.

Mechanism: Instead of representing an integer value $S$ as a sum of $S$ unit spikes over $S$ timesteps, TC-LIF uses temporal weighted firing. It decomposes the integer magnitude into binary bits and assigns weights of $2^{t-1}$ to spikes at timestep $t$ .
Compression: This allows the representation of $L$ $L$ quantization levels using only $T = \log_2(L) - 1$ $T = lo g_{2} (L) - 1$ timesteps (for polar spikes) or $T = \log_2(L)$ $T = lo g_{2} (L)$ (for non-polar).
- Example: For INT4 quantization ( $L=16$ ), standard unfolding requires $T=15$ , whereas TC-LIF reduces this to $T=3$ .
Preservation of Sparsity: Despite the weighted integration, the computation remains spike-driven sparse addition. If a spike bit is 0, the corresponding weight multiplication is bypassed entirely, maintaining the energy efficiency of SNNs.

C. Hardware Co-Design

The authors designed a dedicated RTL accelerator tailored to the SpikeMLLM datapath.

Architecture: The core Processing Element (PE) utilizes a "Spike-driven Dot Product Unit" that replaces dense matrix multiplication with gated accumulation and shift-based temporal weighting.
Optimization: The architecture leverages the reduced temporal depth and multiplier-free nature of TC-LIF to minimize logic area and power consumption.

3. Key Contributions

First Spike-based MLLM Framework: Introduces SpikeMLLM, bridging ANN quantization methods (like QuaRot, GPTQ) with spiking inference.
MSTS: Proposes a modality-aware timestep allocation strategy that improves performance by differentiating temporal scales based on MED, without increasing overall inference cost.
TC-LIF: Develops a temporal compression technique that reduces timesteps from linear ( $L-1$ ) to logarithmic ( $\log_2 L - 1$ ) complexity while preserving representational capacity and spike-driven sparsity.
Algorithm-Hardware Co-Design: Validates the approach with a custom RTL accelerator, demonstrating significant throughput and power gains over GPU baselines.

4. Experimental Results

The framework was evaluated on four mainstream MLLMs (Qwen2VL-7B, InternVL2-8B, MiniCPM-V-2.6-8B, Qwen-VL-Chat-9.6B) and scaled to Qwen2VL-72B.

Performance vs. Baseline: Under aggressive timestep compression ( $T_v/T_t = 3/4$ $T_{v} / T_{t} = 3/4$ ), SpikeMLLM achieves near-lossless performance compared to FP16 baselines.
- InternVL2-8B: Average gap of only 0.72%.
- Qwen2VL-72B: Average gap of only 1.19%.
- Without MSTS/TC-LIF, performance degrades significantly at low timesteps.
Efficiency Gains (Hardware):
- Throughput: The custom accelerator achieved 9.06× higher throughput (393.7 tokens/s) compared to an NVIDIA A800 GPU running FP16.
- Power Efficiency: The accelerator consumed only 7.13 W vs. 184 W for the GPU, resulting in a 25.8× improvement in power efficiency.
Spiking Efficiency: The combination of MSTS and TC-LIF reduced the average firing rate from ~0.53 to ~0.31 and reduced FLOPs by 4.4× compared to standard spike-based methods.

5. Significance

This work represents a major step forward in efficient multimodal intelligence.

Bridging the Gap: It successfully demonstrates that SNNs can scale to large, complex multimodal models without sacrificing accuracy, overcoming the historical limitation of SNNs being restricted to small-scale or unimodal tasks.
Energy Efficiency: By combining algorithmic innovations (MSTS, TC-LIF) with hardware specialization, the paper proves that spike-driven multimodal inference can be orders of magnitude more energy-efficient than current GPU-based solutions.
Future Impact: The results suggest a viable path for deploying advanced AI agents on edge devices and neuromorphic chips, enabling real-time, low-power multimodal reasoning in resource-constrained environments.