Han Xu, Zhiyong Qin, Richard D. Shang, Jiahong Zhang, Xuerui Qiu, Boze Lei, Tiejun Huang, Boxing Xu et al.
We propose the first framework applying the energy efficiency of Spiking Neural Networks (SNNs) to Multimodal Large Language Models (MLLMs). By introducing modality-specific temporal scales and temporal compression techniques, we significantly reduce inference timesteps without performance loss, and achieve high power efficiency with a dedicated hardware accelerator.
Multimodal Large Language Models (MLLMs) achieve remarkable performance but incur substantial computational overhead and energy consumption during inference, limiting deployment in resource-constrained environments. Spiking Neural Networks (SNNs) offer high energy efficiency through event-driven sparse computation, but extending them to MLLMs faces two key challenges: (1) heterogeneous modalities (text, images, etc.) make uniform spike encoding insufficient, and (2) high-resolution image inputs amplify timestep unfolding overhead.
SpikeMLLM unifies existing ANN quantization methods in the spiking representation space, and introduces Modality-Specific Temporal Scales (MSTS) guided by Modality Evolution Discrepancy (MED) and Temporally Compressed LIF (TC-LIF). MSTS adjusts spike encoding according to the temporal characteristics of each modality, and TC-LIF compresses timesteps from T=L-1 to T=log2(L)-1. Additionally, we develop a dedicated RTL accelerator tailored to the spike-driven datapath, implementing algorithm-hardware co-design.
On InternVL2-8B and Qwen2VL-72B, SpikeMLLM maintains near-lossless performance with average gaps of only 0.72% and 1.19% relative to the FP16 baseline. The dedicated RTL accelerator achieves 9.06x higher throughput and 25.8x better power efficiency compared to an FP16 GPU baseline, demonstrating the promise of algorithm-hardware co-design for efficient multimodal intelligence.