Yiqiao Jin, Yiyang Wang, Lucheng Fu, Yijia Xiao, Yinyi Luo, Haoxin Liu, B. Prakash, Josiah D. Hester et al.
UniSD unifies various self-distillation design choices into a systematic framework for efficient LLM adaptation.
Self-distillation in autoregressive LLMs is challenging due to the free-form nature of self-generated trajectories, task-dependent correctness, and unstable supervision from plausible rationales. Existing methods examine isolated design choices, leaving their effectiveness and interactions unclear.
UniSD integrates complementary mechanisms such as multi-teacher agreement, EMA teacher stabilization, token-level contrastive learning, feature matching, and divergence clipping. This systematically addresses supervision reliability, representation alignment, and training stability.
Across six benchmarks and six models from three families, UniSDfull improves over the base model by +5.4 points and the strongest baseline by +2.8 points. The work highlights self-distillation as a practical and steerable approach for efficient LLM adaptation without stronger external teachers.