Bartłomiej Baranowski, Dave Zhenyu Chen, Matthias Nießner
OneCanvas reprojects patch features from multiple views onto a single panoramic canvas for 3D scene understanding, achieving state-of-the-art spatial reasoning with minimal training cost.
Existing VLMs for 3D scene understanding require complex geometry encoders or large training budgets, and struggle with situated reasoning from specific viewpoints.
Each patch is unprojected to 3D world coordinates using depth and camera pose, then placed on a panoramic canvas at the longitude and latitude as seen from the canvas origin, without rasterization or aggregation. A 3D position embedding of metric coordinates is added to restore depth. The pretrained VLM processes this as a normal image. A spatial pretraining curriculum generates on-the-fly supervision by placing object patch features at chosen 3D positions.
OneCanvas achieves state-of-the-art accuracy on SQA3D and VSI-Bench, generalizes to out-of-distribution data on SPBench, and uses an order of magnitude less training compute than competing methods.