arXivGenerative ModelsMethodImportance 8arXiv1mo ago

Alignment Is All You Need For X-to-4D Generation

Qiaowei Miao, Kehan Li, Yawei Luo, Yi Yang

0 CitationsPopularity 1.4

AI analysis

TL;DR

A flexible framework called Align4D is proposed to generate coherent 4D content (video-3D pairs) from any-modal input (X) by aligning it with video and 3D priors.

Problem

Existing 4D generation methods are often modality-specific or require expensive dataset construction for arbitrary user-defined X-to-4D generation, limiting their scalability.

Approach

Object Distance Alignment: Searches for Video-Aligned and Multiview-Aligned Object Distances (VAOD/MAOD) to reconcile 4D renderings with video and multiview diffusion model priors.

Motion-Geometry Joint Alignment: Constrains known and unknown views through synchronized video and 3D inputs to ensure consistent 4D generation.

Asynchronous Optimization: Decouples Gaussian attribute and deformation network training to enhance motion and geometry fidelity.

Results & Contribution

Introduces the X4D dataset for benchmarking. Experiments on X4D and Consistent4D demonstrate that Align4D achieves state-of-the-art quality and consistency in X-to-4D generation.