INoisePredictor<T>

Interfaces · AiDotNet.Interfaces

Interface for noise prediction networks used in diffusion models.

Implements: IFeatureAware, IFullModel<T, Tensor<T>, Tensor<T>>, IGradientComputable<T, Tensor<T>, Tensor<T>>

Known implementations: NoisePredictorBase<T>

For Beginners

A noise predictor is like a “noise detective” that looks at a noisy image and figures out exactly what noise was added to it.

How it works:

  1. The model receives a noisy image and a timestep
  2. The timestep tells the model how much noise should be in the image
  3. The model predicts what noise pattern was added
  4. This prediction is used to remove noise and recover the original image

Different architectures for noise prediction:

  • U-Net: The original and most common, uses an encoder-decoder with skip connections
  • DiT (Diffusion Transformer): Uses transformer blocks, powers state-of-the-art models like SD3 and Sora
  • U-ViT: Hybrid of U-Net and Vision Transformer

The architecture choice affects:

  • Quality of generated images
  • Speed of generation
  • Memory requirements
  • Ability to scale to larger models

How It Works

Noise predictors are the core neural networks in diffusion models that learn to predict the noise added to samples at each timestep. They can be implemented as U-Nets, Diffusion Transformers (DiT), or other architectures.

This interface extends IFullModel to provide all standard model capabilities (training, saving, loading, gradients, checkpointing, etc.).

Properties

Property Summary
bool SupportsCFG { get; } Gets whether this noise predictor supports classifier-free guidance.
bool SupportsCrossAttention { get; } Gets whether this noise predictor supports cross-attention conditioning.
int BaseChannels { get; } Gets the base channel count used in the network architecture.
int ContextDimension { get; } Gets the expected context dimension for cross-attention conditioning.
int InputChannels { get; } Gets the number of input channels the predictor expects.
int OutputChannels { get; } Gets the number of output channels the predictor produces.
int TimeEmbeddingDim { get; } Gets the dimension of the time/timestep embedding.

Methods

Method Summary
Tensor<T> GetTimestepEmbedding(int timestep) Computes the timestep embedding for a given timestep.
Tensor<T> PredictNoise(Tensor<T> noisySample, int timestep, Tensor<T> conditioning) Predicts the noise in a noisy sample at a given timestep.
Tensor<T> PredictNoiseWithEmbedding(Tensor<T> noisySample, Tensor<T> timeEmbedding, Tensor<T> conditioning) Predicts noise with explicit timestep embedding (for batched different timesteps).