SyncDreamerModel<T>
Models & Types · AiDotNet.Diffusion.ThreeD
SyncDreamer model for synchronized multi-view diffusion with 3D-consistent generation.
Inheritance: DiffusionModelBase<T> → LatentDiffusionModelBase<T> → ThreeDDiffusionModelBase<T> → SyncDreamerModel<T>
For Beginners
SyncDreamer creates consistent 3D views from a single image.
How SyncDreamer works:
- Input image is encoded by CLIP into 768-dim features
- 16 viewpoint branches share the same U-Net weights
- 3D-aware attention synchronizes features between all views
- Volume attention ensures spatial consistency across viewpoints
- All 16 views are denoised simultaneously for 3D consistency
- NeuS reconstruction creates a mesh from the consistent views
Key characteristics:
- 16 synchronized views generated simultaneously
- 3D-aware attention prevents inconsistent geometry
- Volume attention for global spatial understanding
- Single-image input to multi-view output
- NeuS-based mesh extraction
When to use SyncDreamer:
- Multi-view consistent image generation
- Single-image 3D reconstruction
- Research on 3D-consistent diffusion
- When view consistency is critical
Limitations:
- Fixed 16-view output configuration
- Higher compute than single-view methods
- Quality limited by SD 1.5 backbone resolution
- NeuS reconstruction can be slow
How It Works
SyncDreamer generates multiple 3D-consistent views simultaneously by synchronizing intermediate features across viewpoints during the diffusion process. A 3D-aware feature attention mechanism and volume attention ensure spatial consistency.
Architecture components:
- Synchronized U-Net with shared weights across 16 viewpoints
- 3D-aware feature attention for cross-view consistency
- Volume attention for spatial understanding across views
- SD 1.5 backbone (320 base channels, 768-dim CLIP)
- NeuS reconstruction from synchronized multi-view outputs
Technical specifications:
- Architecture: Synchronized U-Net with 3D-aware and volume attention
- Backbone: SD 1.5 (320 base channels, [1,2,4,4] multipliers)
- Cross-attention: 768-dim (CLIP)
- Synchronized views: 16
- Reconstruction: NeuS
- Default point count: 4,096
- Scheduler: DDIM
Reference: Liu et al., “SyncDreamer: Generating Multiview-consistent Images from a Single-view Image”, ICLR 2024
Example
var syncDreamer = new SyncDreamerModel<float>();
var points = syncDreamer.GeneratePointCloud(
prompt: "A porcelain teapot",
numPoints: 4096,
numInferenceSteps: 50,
guidanceScale: 7.5);
Constructors
| Constructor | Summary |
|---|---|
SyncDreamerModel(NeuralNetworkArchitecture<T> architecture, DiffusionModelOptions<T> options, INoiseScheduler<T> scheduler, UNetNoisePredictor<T> unet, StandardVAE<T> vae, IConditioningModule<T> conditioner, int defaultPointCount, int? seed) |
Properties
| Property | Summary |
|---|---|
IConditioningModule<T> Conditioner { get; } |
|
INoisePredictor<T> NoisePredictor { get; } |
|
IVAEModel<T> VAE { get; } |
|
bool SupportsMesh { get; } |
|
bool SupportsNovelView { get; } |
|
bool SupportsPointCloud { get; } |
|
bool SupportsScoreDistillation { get; } |
|
bool SupportsTexture { get; } |
|
int LatentChannels { get; } |
|
int NumViews { get; } |
Gets the number of synchronized views generated simultaneously. |
Methods
| Method | Summary |
|---|---|
Mesh3D<T> GenerateMesh(string prompt, string negativePrompt, int resolution, int numInferenceSteps, double guidanceScale, int? seed) |
|
ModelMetadata<T> GetModelMetadata() |
|
Tensor<T> GeneratePointCloud(string prompt, string negativePrompt, int? numPoints, int numInferenceSteps, double guidanceScale, int? seed) |