BrightDreamer: Generic 3D Gaussian Generative Framework for Fast Text-to-3D Synthesis
Arxiv 2024

Interactive Demo

Abstract

Text-to-3D synthesis has recently seen intriguing advances by combining the text-to-image models with 3D representation methods, e.g., Gaussian Splatting (GS), via Score Distillation Sampling (SDS). However, a hurdle of existing methods is the low efficiency, per-prompt optimization for a single 3D object. Therefore, it is imperative for a paradigm shift from per-prompt optimization to one-stage generation for any unseen text prompts, which yet remains challenging. A hurdle is how to directly generate a set of millions of 3D Gaussians to represent a 3D object. This paper presents BrightDreamer, an end-to-end single-stage approach that can achieve generalizable and fast (77 ms) text-to-3D generation. Our key idea is to formulate the generation process as estimating the 3D deformation from an anchor shape with predefined positions. For this, we first propose a Text-guided Shape Deformation (TSD) network to predict the deformed shape and its new positions, used as the centers (one attribute) of 3D Gaussians. To estimate the other four attributes (i.e., scaling, rotation, opacity, and SH coefficient), we then design a novel Text-guided Triplane Generator (TTG) to generate a triplane representation for a 3D object. The center of each Gaussian enables us to transform the triplane feature into the four attributes. The generated 3D Gaussians can be finally rendered at 705 frames per second. Extensive experiments demonstrate the superiority of our method over existing methods. Also, BrightDreamer possesses a strong semantic understanding capability even for complex text prompts.

Overall framework of Our BrightDreamer

Given a text prompt as input, we transform it to a embedding through the frozen CLIP or T5 text encoder. Next, the TSD network (Text-guided Shape Deformation) transforms the fixed anchor positions to the desired shape with text guidance. The new positions are used as the centers of 3D Gaussians. We then design the TTG (Text-guided Triplane Generator) to separately generate three feature planes to construct the implicit spatial representation. Based on the centers of Gaussians, we can obtain their spatial features, which are then transferred to the other attributes through the Gaussian Decoder. Finally, we render 3D Gaussians to 2D images and use the SDS Loss to optimize the whole framework.

BibTeX

             
    @misc{jiang2024brightdreamer,
        title={BrightDreamer: Generic 3D Gaussian Generative Framework for Fast Text-to-3D Synthesis}, 
        author={Lutao Jiang and Lin Wang},
        year={2024},
        eprint={2403.11273},
        archivePrefix={arXiv},
        primaryClass={cs.CV}
    }