DreamGaussianModel<T>
Models & Types · AiDotNet.Diffusion.ThreeD
DreamGaussian model for fast 3D Gaussian splatting generation with Score Distillation Sampling.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → ThreeDDiffusionModelBase<T> → DreamGaussianModel<T>
For Beginners
DreamGaussian creates 3D objects using Gaussian splats in ~2 minutes.
How DreamGaussian works:
- Initialize random 3D Gaussians (position, color, opacity, covariance)
- Render Gaussians from random viewpoints using differentiable rasterization
- Compute SDS loss by comparing renders against the diffusion model’s guidance
- Optimize Gaussian parameters via gradient descent (~500 steps)
- Extract mesh via marching cubes on the Gaussian opacity field
- Refine UV textures using a second SDS stage on the extracted mesh
Key characteristics:
- ~2 minutes total generation (vs hours for NeRF-based methods)
- Gaussian splatting is much faster to optimize than NeRF
- Two-stage: Gaussian optimization + UV texture refinement
- Supports text-to-3D and image-to-3D
- Produces textured meshes ready for rendering
When to use DreamGaussian:
- Fast 3D content prototyping
- Text-to-3D generation with reasonable quality
- Image-to-3D reconstruction
- When speed matters more than maximum quality
Limitations:
- Lower quality than longer optimization methods (DreamFusion, Magic3D)
- Janus problem (multi-face artifacts) from SDS loss
- Mesh quality depends on Gaussian-to-mesh conversion
- Texture refinement is limited by UV unwrapping quality
How It Works
DreamGaussian combines 3D Gaussian Splatting with Score Distillation Sampling (SDS) from a pretrained diffusion model for fast text-to-3D and image-to-3D generation. A second stage refines UV-space textures on the extracted mesh.
Architecture components:
- SD 1.5 U-Net backbone for SDS guidance (320 base channels, 768-dim CLIP)
- 3D Gaussian Splatting as 3D representation
- SDS loss from pretrained 2D diffusion model for 3D optimization
- UV-space texture refinement stage for mesh appearance
- Differentiable rasterization for Gaussian rendering
- Marching cubes mesh extraction from opacity field
Technical specifications:
- 3D representation: 3D Gaussian Splatting (~10,000 Gaussians)
- Guidance backbone: SD 1.5 U-Net (320 base channels, 768-dim CLIP)
- Optimization: ~500 SDS steps for Gaussians + UV refinement
- Total time: ~2 minutes on a single GPU
- Mesh extraction: Marching cubes from opacity field
- Texture: UV-space refinement with SDS
- Noise schedule: Scaled linear beta [0.00085, 0.012], 1000 timesteps
- Scheduler: DDIM
Reference: Tang et al., “DreamGaussian: Generative Gaussian Splatting for Efficient 3D Content Creation”, ICLR 2024
Example
// Create with industry-standard defaults
var dreamGaussian = new DreamGaussianModel<float>();
// Generate 3D point cloud from text
var points = dreamGaussian.GeneratePointCloud(
prompt: "A detailed steampunk robot",
numPoints: 10000,
numInferenceSteps: 500,
guidanceScale: 7.5);
// Generate textured mesh
var mesh = dreamGaussian.GenerateMesh(
prompt: "A cute corgi figurine",
resolution: 256,
numInferenceSteps: 500,
guidanceScale: 7.5);
Constructors
| Constructor | Summary |
|---|---|
DreamGaussianModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, IConditioningModule<T> conditioner, int defaultPointCount, int? seed) |
Initializes a new instance of DreamGaussianModel with full customization support. |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
bool SupportsMesh { get; } |
|
bool SupportsNovelView { get; } |
|
bool SupportsPointCloud { get; } |
|
bool SupportsScoreDistillation { get; } |
|
bool SupportsTexture { get; } |
|
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
Mesh3D<T> GenerateMesh(string prompt, string negativePrompt, int resolution, int numInferenceSteps, double guidanceScale, int? seed) |
|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> GeneratePointCloud(string prompt, string negativePrompt, int? numPoints, int numInferenceSteps, double guidanceScale, int? seed) |