OrbitQuant: Data-Agnostic Quantization for Image and Video Diffusion Transformers

TL;DR

A data-agnostic quantization method for diffusion transformers that addresses shifting activation distributions by operating in a normalized, rotated basis.

Problem

Diffusion transformers have high inference costs, but existing post-training quantization (PTQ) methods require recalibrating for each new checkpoint or modality due to activation distributions shifting across timesteps, prompts, and guidance branches.

Approach

OrbitQuant quantizes in a normalized, rotated basis using a randomized permuted block-Hadamard (RPBH) rotation, concentrating each coordinate around a fixed, known marginal distribution. This allows a single Lloyd-Max codebook to serve all timesteps, prompts, and layers. The rotation is absorbed into weight rows offline, leaving only a forward rotation on activations at runtime.

Results & Contribution

It sets state-of-the-art PTQ performance at several low-bit settings across FLUX.1, Z-Image-Turbo, Wan 2.1, and CogVideoX. It also pushes PTQ of image diffusion transformers to W2A4 with usable generation quality, and the same recipe transfers from image to video with no per-modality tuning.