Megan Frisella, Shubham Tiwari, Andy Ruan, Yi Pan, Parker Gustafson, Mat Jacob, Gilbert Bernstein, Stephanie Wang
Piper is a system that separates distributed training strategies from runtime implementation, enabling users to declaratively define strategies and automatically generate execution plans.
Large-scale model training requires composing multiple parallelism strategies (e.g., data, pipeline, expert parallelism) and memory optimizations like ZeRO. Existing systems either rely on manual strategy design and implementation or support only a fixed set of strategies, making it difficult to integrate new state-of-the-art strategies.
Piper introduces an intermediate representation (IR) that represents the entire training process as a unified global DAG. Users declare a strategy with model annotations and scheduling directives, each of which applies a transformation to the IR. The system then compiles per-device execution plans from the IR and executes them with a strategy-agnostic distributed runtime.
Piper achieves performance parity with existing strategies like ZeRO without degradation, and enables additional performance and memory efficiency gains through joint scheduling of compute and communication in composed parallelism strategies such as DeepSeek-V3's DualPipe.