DiTNoisePredictor<T>

Models & Types · AiDotNet.Diffusion.NoisePredictors

Diffusion Transformer (DiT) noise predictor for diffusion models.

Inheritance: NoisePredictorBase<T> → DiTNoisePredictor<T>

Known derived types: SiTPredictor<T>

For Beginners

DiT is the “new generation” of noise prediction:

Traditional U-Net approach:

  • Uses convolutional neural networks
  • Has encoder-decoder structure with skip connections
  • Good, but limited scalability

DiT approach (this class):

  • Uses transformer architecture (like GPT, but for images)
  • Treats image as patches (like words in a sentence)
  • Scales better with more compute and data
  • Powers cutting-edge models like DALL-E 3, Sora

Key advantages:

  • Better quality at large scales
  • Simpler architecture (no skip connections needed)
  • More flexible conditioning mechanisms
  • Easier to scale training

How It Works

DiT (Diffusion Transformer) replaces the traditional U-Net architecture with a pure transformer design. This approach leverages the scalability and effectiveness of transformers, enabling better performance at larger scales.

Architecture details:

  • Patchify: Split image into 2x2 or larger patches
  • Position embedding: Add spatial information
  • Transformer blocks: Self-attention + MLP
  • AdaLN: Adaptive layer normalization for timestep/conditioning
  • Unpatchify: Reconstruct full resolution output

Used in: DiT (original), DALL-E 3, Sora, SD3, Pixart-alpha

Example

// Create DiT predictor for latent diffusion
var dit = new DiTNoisePredictor<float>(
    inputChannels: 4,      // Latent channels
    hiddenSize: 1152,      // DiT-XL/2 size
    numLayers: 28,         // DiT-XL depth
    numHeads: 16,
    patchSize: 2);
            
// Predict noise
var noisyLatent = new Tensor<float>([1, 4, 32, 32]);      // [batch, latentChannels, h, w]
var timestep = 500;                                          // where we are in the schedule
var textEmbedding = new Tensor<float>([1, 77, 1152]);     // [batch, tokens, hiddenSize]
var noisePrediction = dit.PredictNoise(noisyLatent, timestep, textEmbedding);

Constructors

Constructor Summary
DiTNoisePredictor(NeuralNetworkArchitecture<T> architecture, int inputChannels, int hiddenSize, int numLayers, int numHeads, int patchSize, int contextDim, double mlpRatio, int latentSpatialSize, int numClasses, List<DiTBlock<T>> customBlocks, ILossFunction<T> lossFunction, int? seed) Initializes a new instance of the DiTNoisePredictor class with full customization support.

Properties

Property Summary
bool AreLayersInitialized { get; } Whether this predictor’s lazy weight tensors have been materialized yet (i.e.
bool SupportsBatchedTimeEmbedding { get; }
bool SupportsCFG { get; }
bool SupportsCrossAttention { get; }
int BaseChannels { get; }
int ContextDimension { get; }
int HiddenSize { get; } Gets the hidden size.
int InputChannels { get; }
int NumLayers { get; } Gets the number of layers.
int OutputChannels { get; }
int PatchSize { get; } Gets the patch size.
int TimeEmbeddingDim { get; }

Methods

Method Summary
Tensor<T> PredictNoise(Tensor<T> noisySample, int timestep, Tensor<T> conditioning)
Tensor<T> PredictNoiseWithEmbedding(Tensor<T> noisySample, Tensor<T> timeEmbedding, Tensor<T> conditioning)
void CopyParametersFrom(DiTNoisePredictor<T> source) Copies parameters layer-by-layer from source into this predictor.