DreamFusionModel<T>
Models & Types · AiDotNet.Diffusion.ThreeD
DreamFusion model for text-to-3D generation via Score Distillation Sampling (SDS). Uses a 2D diffusion prior to optimize a 3D neural radiance field representation. Based on “DreamFusion: Text-to-3D using 2D Diffusion” (Poole et al., 2022).
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → DreamFusionModel<T>
For Beginners
Think of DreamFusion as using an AI art critic (the 2D diffusion model) to guide a 3D sculptor (the NeRF). The critic looks at 2D views of the sculpture and gives feedback on how to make it look more like the text description.
How it works:
- You describe what you want: “a DSLR photo of a peacock on a surfboard”
- DreamFusion renders 2D images of a 3D shape from random viewpoints
- The 2D diffusion model evaluates: “Does this look like the prompt?”
- Gradients flow back to improve the 3D representation
- After many iterations, you get a full 3D object you can view from any angle
Key features:
- Creates full 3D assets from text descriptions
- View-consistent: looks correct from any angle
- Leverages the quality of 2D image generators
- No 3D training data required
How It Works
DreamFusion revolutionized text-to-3D generation by using pretrained 2D diffusion models to guide the optimization of a 3D scene representation (NeRF). The key insight is that a 2D diffusion model can serve as a “critic” for 3D content through Score Distillation Sampling.
Technical details:
- Uses NeRF (Neural Radiance Field) for 3D representation
- Employs Score Distillation Sampling (SDS) loss
- Samples random camera views during optimization
- Uses classifier-free guidance with high scale (typically 100)
- Supports mesh extraction via marching cubes
Example
var options = new DiffusionModelOptions<float> { LatentChannels = 4, DefaultInferenceSteps = 50 };
var noise = Tensor<float>.CreateRandom(new[] { 1, 4, 8, 8 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
.ConfigureModel(new DreamFusionModel<float>())
.Build(trainX, trainY);
var rendering = result.Predict(noise);
Constructors
| Constructor | Summary |
|---|---|
DreamFusionModel(NeuralNetworkArchitecture<T> architecture, IDiffusionModel<T> diffusionPrior, DreamFusionConfig config, IConditioningModule<T> conditioner, UNetNoisePredictor<T> unet, int? seed) |
Initializes a new instance of the DreamFusionModel. |
Properties
| Property | Summary |
|---|---|
DreamFusionConfig Config { get; } |
Configuration for DreamFusion model. |
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
int LatentChannels { get; } |
Methods
| Method | Summary |
|---|---|
DreamMesh<T> ExtractMesh(NeRFResult<T> result, int gridResolution, double threshold) |
Generates a mesh from the trained NeRF using marching cubes. |
ModelMetadata<T> GetModelMetadata() |
|
Task<NeRFResult<T>> GenerateAsync(string prompt, int numIterations, double learningRate, double guidanceScale, IProgress<double> progress, CancellationToken cancellationToken) |
Generates a 3D representation from a text prompt using Score Distillation Sampling. |
Tensor<T> PredictNoise(Tensor<T> noisySample, int timestep) |
|
Tensor<T> RenderView(NeRFResult<T> result, CameraPose cameraPose, int resolution) |
Renders an image from the trained NeRF at a specific camera pose. |