MVDreamModel<T>

Models & Types · AiDotNet.Diffusion.ThreeD

MVDream - Multi-View Diffusion Model for 3D-consistent image generation.

Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → ThreeDDiffusionModelBase<T> → MVDreamModel<T>

For Beginners

MVDream creates multiple images of the same object from different angles, all consistent with each other:

Example: “A red sports car”

  • Front view: shows the car’s front grille and headlights
  • Side view: shows the profile with wheels and doors
  • Back view: shows tail lights and rear design
  • All views are consistent (same car, same color, same features)

This consistency is what enables 3D reconstruction:

  • Multiple views + triangulation = 3D model
  • Can be used with Score Distillation for high-quality 3D

How It Works

MVDream is a multi-view diffusion model that generates 3D-consistent images from multiple viewpoints simultaneously. It enables high-quality 3D generation by leveraging multi-view supervision during training.

Key capabilities:

  1. Multi-View Generation: Generate multiple consistent views of an object
  2. Text-to-3D: Create 3D content from text descriptions
  3. Image-to-3D: Convert single images to 3D representations
  4. Score Distillation Sampling (SDS): Guide NeRF/3DGS optimization
  5. Novel View Synthesis: Generate unseen viewpoints of objects

Technical specifications:

  • Image resolution: 256x256 per view (default)
  • Number of views: 4 (orthogonal) or 8 (comprehensive)
  • Latent channels: 4 (Stable Diffusion compatible)
  • Context dimension: 1024 (CLIP/T5 embeddings)
  • Camera model: Spherical coordinates (azimuth, elevation, radius)

Example

var nerFRenderOutput = Tensor<float>.CreateRandom(1, 3, 32, 32);
// Create MVDream model
var mvdream = new MVDreamModel<float>();
            
// Generate 4 views of an object
var views = mvdream.GenerateMultiView(
    prompt: "A cute robot toy",
    numViews: 4,
    numInferenceSteps: 50,
    guidanceScale: 7.5);
            
// Generate for score distillation (SDS)
var sdsGradient = mvdream.ComputeScoreDistillationGradients(
    renderedViews: nerFRenderOutput,
    prompt: "A cute robot toy",
    timestep: 500,
    guidanceScale: 100.0);

Constructors

Constructor Summary
MVDreamModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, MultiViewUNet<T> multiViewUNet, StandardVAE<T> imageVAE, IConditioningModule<T> textConditioner, IConditioningModule<T> imageConditioner, MVDreamConfig config, int? seed) Initializes a new MVDream model with full customization support.

Properties

Property Summary
CameraEmbedding<T> CameraEmbedding { get; } Gets the camera embedding module.
IConditioningModule<T> Conditioner { get; }
IConditioningModule<T> ImageConditioner { get; } Gets the image conditioner for image-to-3D tasks.
INoisePredictor<T> NoisePredictor { get; }
IVAEModel<T> VAE { get; }
MVDreamConfig Config { get; } Gets the model configuration.
bool SupportsMesh { get; }
bool SupportsNovelView { get; }
bool SupportsPointCloud { get; }
bool SupportsScoreDistillation { get; }
bool SupportsTexture { get; }
int LatentChannels { get; }

Methods

Method Summary
Mesh3D<T> ImageTo3D(Tensor<T> inputImage, int numViews, int numInferenceSteps, double guidanceScale, int? seed) Generates 3D from a single input image.
Tensor<T> ComputeScoreDistillationGradients(Tensor<T> renderedViews, string prompt, int timestep, double guidanceScale) Computes Score Distillation Sampling gradients for 3D optimization.
Tensor<T>[] GenerateMultiView(string prompt, string negativePrompt, int numViews, int numInferenceSteps, double guidanceScale, double elevation, int? seed) Generates multiple consistent views from a text prompt.
Tensor<T>[] GenerateNovelViewsFromImage(Tensor<T> inputImage, int numViews, int numInferenceSteps, double guidanceScale, int? seed) Generates novel views from a single input image.

Fields

Field Summary
double DEFAULT_CAMERA_DISTANCE Default camera distance from object.
int MVDREAM_BASE_CHANNELS MVDream base channels for U-Net.
int MVDREAM_CONTEXT_DIM Context dimension for text/image conditioning.
int MVDREAM_IMAGE_SIZE Default image resolution for each view.
int MVDREAM_LATENT_CHANNELS MVDream latent channels (Stable Diffusion compatible).