UViTNoisePredictor<T>
Models & Types · AiDotNet.Diffusion.NoisePredictors
U-shaped Vision Transformer (U-ViT) noise predictor for diffusion models.
Inheritance: NoisePredictorBase<T> → UViTNoisePredictor<T>
For Beginners
U-ViT is like a transformer with U-Net-style shortcuts:
U-Net: Uses conv layers with skip connections DiT: Uses transformer layers without skip connections U-ViT: Uses transformer layers WITH skip connections (best of both)
Architecture:
- Patchify: Split image into patches
- Encoder transformer blocks (L/2 blocks)
- Middle transformer block
- Decoder transformer blocks (L/2 blocks) with skip connections from encoder
- Unpatchify: Reconstruct output
Used in: UniDiffuser (multi-modal generation).
How It Works
U-ViT combines the best of U-Net and Vision Transformer architectures. It applies a transformer to image patches but adds long skip connections between encoder and decoder blocks, similar to U-Net.
Reference: Bao et al., “All are Worth Words: A ViT Backbone for Diffusion Models”, CVPR 2023
Example
var predictor = new UViTNoisePredictor<float>(inputChannels: 4, hiddenSize: 1024, numLayers: 22, numHeads: 16);
var noisyLatent = Tensor<float>.CreateRandom(new[] { 1, 4, 32, 32 });
var predicted = predictor.PredictNoise(noisyLatent, timestep: 500);
Constructors
| Constructor | Summary |
|---|---|
UViTNoisePredictor(int inputChannels, int hiddenSize, int numLayers, int numHeads, int patchSize, int contextDim, int latentSpatialSize, int? seed) |
Initializes a new instance of the U-ViT noise predictor. |
Properties
| Property | Summary |
|---|---|
bool SupportsBatchedTimeEmbedding { get; } |
|
bool SupportsCFG { get; } |
|
bool SupportsCrossAttention { get; } |
|
int BaseChannels { get; } |
|
int ContextDimension { get; } |
|
int HiddenSize { get; } |
Gets the hidden dimension of the transformer. |
int InputChannels { get; } |
|
int OutputChannels { get; } |
|
int PatchSize { get; } |
Gets the patch size used for tokenizing spatial features. |
int TimeEmbeddingDim { get; } |
Methods
| Method | Summary |
|---|---|
Tensor<T> PredictNoise(Tensor<T> noisySample, int timestep, Tensor<T> conditioning) |
|
Tensor<T> PredictNoiseWithEmbedding(Tensor<T> noisySample, Tensor<T> timeEmbedding, Tensor<T> conditioning) |