Xiwen Chen, Jingjing Wang, Wenhui Zhu, Peijie Qiu, Xuanzhao Dong, Hejian Sang, Zhipeng Wang, A. Geramifard et al.
SODA is a semi on-policy black-box distillation method that contrasts the student's static outputs with the teacher's optimal response, significantly reducing the instability and computational cost of existing on-policy methods while achieving superior performance.
In black-box LLM distillation, off-policy methods (e.g., sequence-level KD) struggle to correct the student's inherent errors, while fully on-policy methods (e.g., GAD) suffer from training instability due to adversarial training and crippling computational overhead.
Leveraging the fact that a compact student model's zero-shot responses are almost always inferior to the teacher's responses, we construct a contrastive signal by pairing the teacher's optimal response with a one-time static snapshot of the student's outputs. This enables high-quality distribution alignment without costly dynamic rollouts or fragile adversarial balancing.
Experiments on four compact Qwen2.5 and Llama-3 models show that SODA matches or outperforms state-of-the-art methods on 15 out of 16 benchmarks. It achieves 10x faster training, 27% less peak GPU memory usage, and completely eliminates adversarial instability.