UNetNoisePredictor<T>

Models & Types · AiDotNet.Diffusion.NoisePredictors

U-Net architecture for noise prediction in diffusion models.

Inheritance: NoisePredictorBase<T> → UNetNoisePredictor<T>

For Beginners

Think of U-Net like a funnel:

  1. Encoder (going down): Compresses the image, capturing patterns at different scales
  2. Middle: Processes the most compressed representation
  3. Decoder (going up): Expands back to original size, using skip connections

Skip connections are like “shortcuts” that connect encoder layers directly to decoder layers, helping the network preserve fine details that might otherwise be lost during compression.

How It Works

The U-Net is the most common architecture for diffusion model noise prediction. It has an encoder-decoder structure with skip connections that help preserve fine-grained details during the denoising process.

This implementation follows the Stable Diffusion architecture with:

  • Residual blocks with group normalization
  • Self-attention at lower resolutions
  • Cross-attention for text conditioning
  • Time embedding injection via adaptive normalization

Example

var predictor = new UNetNoisePredictor<float>(inputChannels: 4, baseChannels: 320, contextDim: 768);
var noisyLatent = Tensor<float>.CreateRandom(new[] { 1, 4, 64, 64 });
var predicted = predictor.PredictNoise(noisyLatent, timestep: 500);

Constructors

Constructor Summary
UNetNoisePredictor(NeuralNetworkArchitecture<T> architecture, int inputChannels, int? outputChannels, int baseChannels, int[] channelMultipliers, int numResBlocks, int[] attentionResolutions, int contextDim, int numHeads, int inputHeight, List<UNetBlock<T>> encoderBlocks, List<UNetBlock<T>> middleBlocks, List<UNetBlock<T>> decoderBlocks, ILossFunction<T> lossFunction, int? seed) Initializes a new instance of the UNetNoisePredictor class with full customization support.

Properties

Property Summary
bool SupportsBatchedTimeEmbedding { get; }
bool SupportsCFG { get; }
bool SupportsCrossAttention { get; }
int BaseChannels { get; }
int ContextDimension { get; }
int InputChannels { get; }
int OutputChannels { get; }
int TimeEmbeddingDim { get; }
int[] ChannelMultipliers { get; } Gets the per-level channel multipliers of this UNet’s encoder/decoder.

Methods

Method Summary
Tensor<T> PredictNoise(Tensor<T> noisySample, int timestep, Tensor<T> conditioning)
Tensor<T> PredictNoiseWithEmbedding(Tensor<T> noisySample, Tensor<T> timeEmbedding, Tensor<T> conditioning)
bool CompileForward(Tensor<T> sampleNoisy, int sampleTimestep, Tensor<T> conditioning) Eagerly compiles the UNet forward pass for the given sample input shape, storing the plan in the per-instance cache.
int DecorateAttentionBlocks(Func<ILayer<T>, int, bool, ILayer<T>> decorator) Replaces every self- and cross-attention block with the result of decorator, so an adapter (LoRA and friends) can be injected without this class knowing about it.