Haofei Xu, Rundi Wu, Philipp Henzler, Nikolai Kalischek, Michael Oechsle, Fabian Manhardt, Marc Pollefeys, Andreas Geiger et al.
This work introduces PointDiT, a minimalist pixel-space Diffusion Transformer for monocular geometry estimation that eliminates the architectural overhead of existing methods, achieving superior performance with a simpler design.
State-of-the-art single-image 3D reconstruction methods often rely on complex hybrid architectures, intricate loss functions, or compress geometry into latent spaces. This introduces unnecessary architectural overhead and can limit performance, especially in ambiguous regions.
The authors propose a plain Vision Transformer (ViT)-based diffusion model operating directly on raw 3D point map patches. It is conditioned on image tokens from a pre-trained DINOv3 and trained entirely from scratch, eliminating the need for point map tokenizers.
Despite its simplicity, PointDiT surpasses complex latent-based diffusion models and is significantly simpler than hybrid alternatives. It produces sharper geometric structures and demonstrates greater robustness in highly ambiguous regions, such as transparent objects, challenging the need for complex architectural designs in this domain.