UViTNoisePredictor<T>

Models & Types · AiDotNet.Diffusion.NoisePredictors

U-shaped Vision Transformer (U-ViT) noise predictor for diffusion models.

Inheritance: NoisePredictorBase<T> → UViTNoisePredictor<T>

For Beginners

U-ViT is like a transformer with U-Net-style shortcuts:

U-Net: Uses conv layers with skip connections DiT: Uses transformer layers without skip connections U-ViT: Uses transformer layers WITH skip connections (best of both)

Architecture:

  1. Patchify: Split image into patches
  2. Encoder transformer blocks (L/2 blocks)
  3. Middle transformer block
  4. Decoder transformer blocks (L/2 blocks) with skip connections from encoder
  5. Unpatchify: Reconstruct output

Used in: UniDiffuser (multi-modal generation).

How It Works

U-ViT combines the best of U-Net and Vision Transformer architectures. It applies a transformer to image patches but adds long skip connections between encoder and decoder blocks, similar to U-Net.

Reference: Bao et al., “All are Worth Words: A ViT Backbone for Diffusion Models”, CVPR 2023

Example

var predictor = new UViTNoisePredictor<float>(inputChannels: 4, hiddenSize: 1024, numLayers: 22, numHeads: 16);
var noisyLatent = Tensor<float>.CreateRandom(new[] { 1, 4, 32, 32 });
var predicted = predictor.PredictNoise(noisyLatent, timestep: 500);

Constructors

Constructor Summary
UViTNoisePredictor(int inputChannels, int hiddenSize, int numLayers, int numHeads, int patchSize, int contextDim, int latentSpatialSize, int? seed) Initializes a new instance of the U-ViT noise predictor.

Properties

Property Summary
bool SupportsBatchedTimeEmbedding { get; }
bool SupportsCFG { get; }
bool SupportsCrossAttention { get; }
int BaseChannels { get; }
int ContextDimension { get; }
int HiddenSize { get; } Gets the hidden dimension of the transformer.
int InputChannels { get; }
int OutputChannels { get; }
int PatchSize { get; } Gets the patch size used for tokenizing spatial features.
int TimeEmbeddingDim { get; }

Methods

Method Summary
Tensor<T> PredictNoise(Tensor<T> noisySample, int timestep, Tensor<T> conditioning)
Tensor<T> PredictNoiseWithEmbedding(Tensor<T> noisySample, Tensor<T> timeEmbedding, Tensor<T> conditioning)