ShortGPT: Layers in Large Language Models are More Redundant Than You Expect

TL;DR

This study shows that many layers in LLMs are redundant, and simply removing layers based on BI scores enables effective model compression.

Problem

The rapid increase in LLM size leads to high inference costs and memory requirements. Existing pruning methods often require complex structures or fine-tuning, and lack systematic analysis of layer-level redundancy.

Approach

We analyze layer similarity across multiple LLMs and find that many layers are similar and some have negligible influence. We define the Block Influence (BI) metric to measure each layer's importance, and propose ShortGPT, which sequentially removes layers with low BI scores without additional training or complex reconstruction.

Results & Contribution

ShortGPT outperforms previous state-of-the-art pruning methods on various LLMs such as LLaMA and OPT. It is orthogonal to quantization methods, and demonstrates that simple layer removal is more effective than complex techniques, suggesting high redundancy in LLM architectures.