Optimizing generative AI by backpropagating language model feedback

TL;DR

A novel training method that optimizes generative AI models by backpropagating language model feedback.

Problem

Improving the output quality of generative AI models typically requires human feedback, which is costly and lacks scalability. While language models can serve as evaluators, directly using their scores for training generative models is challenging.

Approach

The proposed method converts scores assigned by a language model (e.g., usefulness, accuracy) into a differentiable form and uses them as the loss function for the generative model. This allows the generative model to receive direct backpropagation of language model feedback. Specifically, scores are approximated using the output probabilities of the language model, and gradients with respect to the generative model's parameters are computed.

Results & Contribution

Significant quality improvements are demonstrated in text and image generation tasks without human feedback. Compared to existing reinforcement learning-based methods, the proposed approach achieves higher sample efficiency and more stable training. This method is applicable to various generative models and offers a practical alternative that reduces the burden on human evaluators.