DiTNoisePredictor<T>
Models & Types · AiDotNet.Diffusion.NoisePredictors
Diffusion Transformer (DiT) noise predictor for diffusion models.
Inheritance: NoisePredictorBase<T> → DiTNoisePredictor<T>
Known derived types: SiTPredictor<T>
For Beginners
DiT is the “new generation” of noise prediction:
Traditional U-Net approach:
- Uses convolutional neural networks
- Has encoder-decoder structure with skip connections
- Good, but limited scalability
DiT approach (this class):
- Uses transformer architecture (like GPT, but for images)
- Treats image as patches (like words in a sentence)
- Scales better with more compute and data
- Powers cutting-edge models like DALL-E 3, Sora
Key advantages:
- Better quality at large scales
- Simpler architecture (no skip connections needed)
- More flexible conditioning mechanisms
- Easier to scale training
How It Works
DiT (Diffusion Transformer) replaces the traditional U-Net architecture with a pure transformer design. This approach leverages the scalability and effectiveness of transformers, enabling better performance at larger scales.
Architecture details:
- Patchify: Split image into 2x2 or larger patches
- Position embedding: Add spatial information
- Transformer blocks: Self-attention + MLP
- AdaLN: Adaptive layer normalization for timestep/conditioning
- Unpatchify: Reconstruct full resolution output
Used in: DiT (original), DALL-E 3, Sora, SD3, Pixart-alpha
Example
// Create DiT predictor for latent diffusion
var dit = new DiTNoisePredictor<float>(
inputChannels: 4, // Latent channels
hiddenSize: 1152, // DiT-XL/2 size
numLayers: 28, // DiT-XL depth
numHeads: 16,
patchSize: 2);
// Predict noise
var noisyLatent = new Tensor<float>([1, 4, 32, 32]); // [batch, latentChannels, h, w]
var timestep = 500; // where we are in the schedule
var textEmbedding = new Tensor<float>([1, 77, 1152]); // [batch, tokens, hiddenSize]
var noisePrediction = dit.PredictNoise(noisyLatent, timestep, textEmbedding);
Constructors
| Constructor | Summary |
|---|---|
DiTNoisePredictor(NeuralNetworkArchitecture<T> architecture, int inputChannels, int hiddenSize, int numLayers, int numHeads, int patchSize, int contextDim, double mlpRatio, int latentSpatialSize, int numClasses, List<DiTBlock<T>> customBlocks, ILossFunction<T> lossFunction, int? seed) |
Initializes a new instance of the DiTNoisePredictor class with full customization support. |
Properties
| Property | Summary |
|---|---|
bool AreLayersInitialized { get; } |
Whether this predictor’s lazy weight tensors have been materialized yet (i.e. |
bool SupportsBatchedTimeEmbedding { get; } |
|
bool SupportsCFG { get; } |
|
bool SupportsCrossAttention { get; } |
|
int BaseChannels { get; } |
|
int ContextDimension { get; } |
|
int HiddenSize { get; } |
Gets the hidden size. |
int InputChannels { get; } |
|
int NumLayers { get; } |
Gets the number of layers. |
int OutputChannels { get; } |
|
int PatchSize { get; } |
Gets the patch size. |
int TimeEmbeddingDim { get; } |
Methods
| Method | Summary |
|---|---|
Tensor<T> PredictNoise(Tensor<T> noisySample, int timestep, Tensor<T> conditioning) |
|
Tensor<T> PredictNoiseWithEmbedding(Tensor<T> noisySample, Tensor<T> timeEmbedding, Tensor<T> conditioning) |
|
void CopyParametersFrom(DiTNoisePredictor<T> source) |
Copies parameters layer-by-layer from source into this predictor. |