DreamFusionModel<T>

Models & Types · AiDotNet.Diffusion.ThreeD

DreamFusion model for text-to-3D generation via Score Distillation Sampling (SDS). Uses a 2D diffusion prior to optimize a 3D neural radiance field representation. Based on “DreamFusion: Text-to-3D using 2D Diffusion” (Poole et al., 2022).

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → DreamFusionModel<T>

For Beginners

Think of DreamFusion as using an AI art critic (the 2D diffusion model) to guide a 3D sculptor (the NeRF). The critic looks at 2D views of the sculpture and gives feedback on how to make it look more like the text description.

How it works:

  1. You describe what you want: “a DSLR photo of a peacock on a surfboard”
  2. DreamFusion renders 2D images of a 3D shape from random viewpoints
  3. The 2D diffusion model evaluates: “Does this look like the prompt?”
  4. Gradients flow back to improve the 3D representation
  5. After many iterations, you get a full 3D object you can view from any angle

Key features:

  • Creates full 3D assets from text descriptions
  • View-consistent: looks correct from any angle
  • Leverages the quality of 2D image generators
  • No 3D training data required

How It Works

DreamFusion revolutionized text-to-3D generation by using pretrained 2D diffusion models to guide the optimization of a 3D scene representation (NeRF). The key insight is that a 2D diffusion model can serve as a “critic” for 3D content through Score Distillation Sampling.

Technical details:

  • Uses NeRF (Neural Radiance Field) for 3D representation
  • Employs Score Distillation Sampling (SDS) loss
  • Samples random camera views during optimization
  • Uses classifier-free guidance with high scale (typically 100)
  • Supports mesh extraction via marching cubes

Example

var options = new DiffusionModelOptions<float> { LatentChannels = 4, DefaultInferenceSteps = 50 };
var noise = Tensor<float>.CreateRandom(new[] { 1, 4, 8, 8 });
var trainX = Tensor<float>.CreateRandom(4, 8);
var trainY = Tensor<float>.CreateRandom(4, 2);
var result = new AiModelBuilder<float, Tensor<float>, Tensor<float>>()
    .ConfigureModel(new DreamFusionModel<float>())
    .Build(trainX, trainY);
var rendering = result.Predict(noise);

Constructors

Constructor Summary
DreamFusionModel(NeuralNetworkArchitecture<T> architecture, IDiffusionModel<T> diffusionPrior, DreamFusionConfig config, IConditioningModule<T> conditioner, UNetNoisePredictor<T> unet, int? seed) Initializes a new instance of the DreamFusionModel.

Properties

Property Summary
DreamFusionConfig Config { get; } Configuration for DreamFusion model.
IConditioningModule<T> Conditioner { get; }
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
int LatentChannels { get; }

Methods

Method Summary
DreamMesh<T> ExtractMesh(NeRFResult<T> result, int gridResolution, double threshold) Generates a mesh from the trained NeRF using marching cubes.
ModelMetadata<T> GetModelMetadata()
Task<NeRFResult<T>> GenerateAsync(string prompt, int numIterations, double learningRate, double guidanceScale, IProgress<double> progress, CancellationToken cancellationToken) Generates a 3D representation from a text prompt using Score Distillation Sampling.
Tensor<T> PredictNoise(Tensor<T> noisySample, int timestep)
Tensor<T> RenderView(NeRFResult<T> result, CameraPose cameraPose, int resolution) Renders an image from the trained NeRF at a specific camera pose.