Jen-Hao Cheng, Yipeng Wang, Hao Zhang, Gengshan Yang, Jenq-Neng Hwang
Flex4DHuman is a diffusion model that generates dense multi-view videos from monocular or sparse multi-view videos using only relative camera-pose conditioning, and reconstructs them into 4D Gaussian splats.
Reconstructing dynamic humans in 4D requires synchronized multi-view videos, but real-world settings often involve only a single camera or a few cameras. Existing methods rely on explicit geometric priors like skeletons, depth maps, or normals, and are limited to specific domains, lacking generalization.
Flex4DHuman builds on the Wan 2.1 1.3B text-to-video model, conditioning on relative camera poses via a five-axis positional encoding (extending spatio-temporal RoPE with view indices and SE(3) relative geometry). It is trained with a three-stage curriculum: pose following, flexible reference-to-target view generation, and temporal rollout. During temporal rollout, clean historical target-view tokens are used. Multi-view captions are added for test-time text control.
Flex4DHuman surpasses prior state-of-the-art methods on DNA-Rendering and ActorsHQ datasets, and generalizes to animal categories after mixed human-animal training. It enables 4D content creation from casual monocular videos, contributing practically to simulation, gaming, AR/VR, and video re-shooting.