Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Lifeng Dong, Ruiping Wang et al.
BitNet b1.58 quantizes all weights to 1.58 bits (ternary values), enabling extremely efficient LLM inference while maintaining FP16-level performance.
Large Language Models (LLMs) consume massive memory and energy, leading to high deployment costs and difficulty in use on edge devices. Existing quantization methods often suffer from performance degradation.
BitNet b1.58 introduces 1.58-bit quantization that represents weights as only three values: -1, 0, and 1. It uses quantization-aware training during training to minimize performance loss, and employs matrix multiplication kernels optimized for ternary weights to maximize inference speed and energy efficiency.
With the same model size and number of training tokens, it matches FP16 models in perplexity and downstream task performance. Latency is improved by 2-4x, memory usage by 3-4x, throughput by 2-4x, and energy consumption by 5-10x. This ushers in the era of 1-bit LLMs and presents a new paradigm for dedicated hardware design.