Fanxu Meng, Zhaohui Wang, Muhan Zhang
PiSSA replaces LoRA's adapter initialization with the principal singular values and singular vectors of the weight matrix, updating only the principal components during fine-tuning while freezing the rest, thereby improving convergence speed and performance.
LoRA learns adapters initialized with Gaussian noise and zeros, which can lead to slow convergence and limited final performance due to random initialization. Additionally, it fails to fully leverage the important information (principal components) of the pre-trained weights.
PiSSA performs SVD on the original weight matrix W to extract the top r singular values and singular vectors, and uses them to initialize the adapter matrices A and B. The remaining singular values and vectors are separated into a frozen residual matrix W_res. During fine-tuning, only A and B are updated while W_res remains fixed. This maintains the same parameter count and architecture as LoRA, but enables direct learning of the principal components for more efficient adaptation.
Consistent performance improvements over LoRA were observed across 12 models (184M~70B) and 13 NLG/NLU tasks. On GSM8K, Mistral-7B fine-tuned with PiSSA achieved 72.86% accuracy, surpassing LoRA's 67.7%. QPiSSA, combined with QLoRA, achieved 86.05% on LLaMA-3-70B, outperforming QLoRA's 81.73%. Using a fast SVD technique, initialization can be done in seconds, making the transition from LoRA nearly cost-free.