VideoUNetPredictor<T>
Models & Types ยท AiDotNet.Diffusion.NoisePredictors
3D U-Net architecture for video noise prediction in diffusion models.
Inheritance: NoisePredictorBase<T> โ VideoUNetPredictor<T>
For Beginners
While a regular U-Net processes single images, VideoUNet processes sequences of frames as a 3D volume:
Regular U-Net:
- Input: [batch, channels, height, width]
- 2D convolutions across spatial dimensions only
- Each image processed independently
Video U-Net:
- Input: [batch, channels, frames, height, width]
- 3D convolutions across space AND time
- Frames are processed together, understanding motion
Key features:
- Temporal convolutions capture motion patterns
- Temporal attention for long-range frame relationships
- Skip connections across both space and time
- Image conditioning for image-to-video generation
Used in: Stable Video Diffusion, ModelScope, VideoCrafter
How It Works
The VideoUNetPredictor extends the standard U-Net architecture to handle video data by incorporating 3D convolutions and temporal attention. This is the core noise prediction network used in video diffusion models like Stable Video Diffusion.
Architecture details:
- Encoder: 3D ResBlocks with temporal + spatial attention
- Middle: Multiple 3D attention blocks
- Decoder: 3D ResBlocks with skip connections
- Temporal convolutions with kernel size 3 across frames
Example
var predictor = new VideoUNetPredictor<float>(inputChannels: 4, baseChannels: 320, numFrames: 14);
var noisyVideo = Tensor<float>.CreateRandom(new[] { 1, 4, 14, 64, 64 });
var predicted = predictor.PredictNoise(noisyVideo, timestep: 500);
Constructors
| Constructor | Summary |
|---|---|
VideoUNetPredictor(int inputChannels, int? outputChannels, int baseChannels, int[] channelMultipliers, int numResBlocks, int[] attentionResolutions, int numTemporalLayers, int contextDim, int numHeads, bool supportsImageConditioning, int inputHeight, int inputWidth, int numFrames, int clipTokenLength, ILossFunction<T> lossFunction, int? seed, int? imageConditionChannels, bool concatenateImageCondition, int numClassEmbeddings, VideoUNetArchitectureProfile architectureProfile) |
Initializes a new instance of the VideoUNetPredictor class. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<ILayer<T>> TemporalTrainingLayers { get; } |
Gets the inserted temporal layers while excluding pretrained spatial U-Net layers. |
IReadOnlyList<int> ChannelMultipliers { get; } |
Gets the paper-defined channel multiplier at each U-Net resolution. |
VideoUNetArchitectureProfile ArchitectureProfile { get; } |
Gets the selected topology profile. |
bool ConcatenatesImageCondition { get; } |
Gets whether conditioning is concatenated before the input convolution. |
bool SupportsBatchedTimeEmbedding { get; } |
|
bool SupportsCFG { get; } |
|
bool SupportsCrossAttention { get; } |
|
bool SupportsImageConditioning { get; } |
Gets whether this predictor supports image conditioning for image-to-video. |
bool UsesTemporalTransformerAttention { get; } |
Gets whether any temporal transformer-attention layer is present. |
int BaseChannels { get; } |
|
int ContextDimension { get; } |
|
int DownsampleCount { get; } |
Gets the number of spatial downsampling operations. |
int ImageConditionChannels { get; } |
Gets the expected image/video condition channel count. |
int InputChannels { get; } |
|
int NumClassEmbeddings { get; } |
Gets the number of learned degradation/noise-level embeddings. |
int NumResBlocks { get; } |
Gets the number of spatial residual layers in each down block. |
int NumTemporalLayers { get; } |
Gets the number of temporal transformer layers. |
int OnlyCrossAttentionTransformerCount { get; } |
Gets the number of Transformer3D blocks whose first attention is text-only. |
int OutputChannels { get; } |
|
int SpatialResBlockCount { get; } |
Gets the number of spatial residual blocks in the materialized topology. |
int TemporalModuleCount { get; } |
Gets the number of explicit temporal modules in the materialized topology. |
int TimeEmbeddingDim { get; } |
|
int UpsampleCount { get; } |
Gets the number of spatial upsampling operations. |
int VideoTransformerCount { get; } |
Gets the number of released Transformer3D blocks. |
Methods
| Method | Summary |
|---|---|
Tensor<T> PredictNoise(Tensor<T> noisySample, int timestep, Tensor<T> conditioning) |
|
Tensor<T> PredictNoiseWithEmbedding(Tensor<T> noisySample, Tensor<T> timeEmbedding, Tensor<T> conditioning) |
|
Tensor<T> PredictNoiseWithImageCondition(Tensor<T> noisySample, int timestep, Tensor<T> imageCondition, Tensor<T> textConditioning) |
Predicts noise for image-to-video generation with image conditioning. |
Tensor<T> PredictNoiseWithVideoCondition(Tensor<T> noisySample, int timestep, Tensor<T> videoCondition, Tensor<T> textConditioning, int? noiseLevel) |
Predicts noise while conditioning every frame on its corresponding low-resolution video latent. |
Vector<T> GetParameters() |
|
void SetParameters(Vector<T> parameters) |