Dream 7B: Diffusion Large Language Models

TL;DR

Dream 7B is the most powerful open diffusion large language model using discrete diffusion modeling, enabling parallel processing and flexible inference compared to autoregressive models.

Problem

Autoregressive language models suffer from slow inference due to sequential generation, constraints on generation order, and difficulty with infilling or quality-speed trade-offs. Diffusion models can address these issues, but previous diffusion language models significantly underperform autoregressive models.

Approach

Dream 7B iteratively refines noisy token sequences through a discrete diffusion process for parallel generation. Key techniques include AR-based LLM initialization (initializing from an autoregressive language model) and context-adaptive token-level noise rescheduling, which greatly improve diffusion model performance.

Results & Contribution

Dream 7B consistently outperforms existing diffusion language models on general, mathematical, and coding tasks, achieving performance comparable to autoregressive models. It also enables arbitrary-order generation, infilling, and tunable quality-speed trade-offs during inference, significantly enhancing the practicality of diffusion language models. The model is open-sourced to facilitate further research.