TL;DR
This paper proposes a framework that extends single-image 3D asset generators to the scene scale, synthesizing large, controllable 3D scenes from a prompt-generated scene image.
Problem
Existing image-to-3D generators excel at creating single assets but struggle to scale to full scene synthesis involving multiple objects and complex spatial relationships. A major bottleneck is the scarcity of 3D scene-level data for training or fine-tuning such models.
Approach
Synthetic Data Engine: To address data scarcity, the authors propose a new engine that automatically generates scene layouts by composing various 3D assets, creating synthetic scene-level training data.
Adaptation as a Convolutional Operator: The base image-to-3D generator is fine-tuned on this synthetic scene data to function as a convolutional operator. It takes a dimetric image of an entire scene as input and generates the corresponding 3D structure for that region.
Scene Synthesis Pipeline: A user prompt is used to generate a dimetric image of the entire scene. This image is then processed by the convolutional operator to synthesize a final 3D scene of arbitrary size and complexity.
Results & Contribution
The method successfully generates large, coherent, and detailed 3D scenes across diverse prompts and layouts, overcoming limitations of prior approaches.
It introduces a novel synthetic data engine to tackle the core problem of scarce 3D scene data, offering a valuable data generation methodology for future research.