MMDiTNoisePredictor<T>
Models & Types · AiDotNet.Diffusion.NoisePredictors
Multi-Modal Diffusion Transformer (MMDiT) noise predictor for SD3 and FLUX architectures.
Inheritance: NoisePredictorBase<T> → MMDiTNoisePredictor<T>
Known derived types: AsymmDiTPredictor<T>, EMMDiTPredictor<T>, FluxDoubleStreamPredictor<T>, MMDiTXNoisePredictor<T>
For Beginners
MMDiT is the architecture behind SD3 and FLUX:
How MMDiT differs from standard DiT:
- Standard DiT: Image patches processed by transformer, text injected via cross-attention
- MMDiT: Image AND text tokens are concatenated and processed together through JOINT attention
Key characteristics:
- Joint attention: Both text and image tokens attend to each other equally
- Separate MLPs: Text and image tokens have independent MLP layers after joint attention
- Dual stream: Text and image streams with shared attention but separate feed-forward
- AdaLN-Zero: Adaptive layer normalization with zero-init gating
- Supports multiple text encoders (CLIP + T5 for SD3, CLIP + T5 for FLUX)
Used in:
- Stable Diffusion 3 / SD 3.5 (Stability AI)
- FLUX.1 dev/schnell/pro (Black Forest Labs)
- Pixart-Sigma
Advantages over standard DiT:
- Better text-image alignment through joint attention
- More expressive conditioning without cross-attention bottleneck
- Scales better with model size
How It Works
MMDiT extends the standard DiT architecture by processing text and image tokens jointly through shared transformer blocks, rather than using cross-attention. This enables deeper bidirectional interaction between modalities.
Technical specifications:
- Architecture: Multi-modal transformer with joint self-attention
- SD3 Medium: 2B params, 24 layers, hidden 1536, 24 heads
- FLUX.1 dev: ~12B params, 19 double + 38 single layers, hidden 3072, 24 heads
- Patch size: 2 (latent space)
- Conditioning: Concatenated CLIP + T5 text embeddings
- Timestep: Sinusoidal + MLP projection
- Positional encoding: RoPE (Rotary Position Embedding)
Reference: Esser et al., “Scaling Rectified Flow Transformers for High-Resolution Image Synthesis”, ICML 2024
Example
// Create MMDiT for SD3
var mmdit = new MMDiTNoisePredictor<float>(
inputChannels: 16, // SD3 uses 16 latent channels
hiddenSize: 1536, // SD3 Medium
numJointLayers: 24,
numHeads: 24,
contextDim: 4096); // T5-XXL
// Create MMDiT for FLUX
var mmdit2 = new MMDiTNoisePredictor<float>(
inputChannels: 16,
hiddenSize: 3072, // FLUX.1
numJointLayers: 19,
numSingleLayers: 38,
numHeads: 24,
contextDim: 4096);
Constructors
| Constructor | Summary |
|---|---|
MMDiTNoisePredictor(NeuralNetworkArchitecture<T> architecture, int inputChannels, int hiddenSize, int numJointLayers, int numSingleLayers, int numHeads, int patchSize, int contextDim, double mlpRatio, List<MMDiTBlock<T>> customJointBlocks, List<MMDiTSingleBlock<T>> customSingleBlocks, ILossFunction<T> lossFunction, int? seed) |
Initializes a new instance of the MMDiTNoisePredictor class with full customization support. |
Properties
| Property | Summary |
|---|---|
bool SupportsCFG { get; } |
|
bool SupportsCrossAttention { get; } |
|
int BaseChannels { get; } |
|
int ContextDimension { get; } |
|
int HiddenSize { get; } |
Gets the hidden size of the transformer. |
int InputChannels { get; } |
|
int NumJointLayers { get; } |
Gets the number of joint (double-stream) transformer layers. |
int NumSingleLayers { get; } |
Gets the number of single-stream transformer layers (FLUX-style). |
int OutputChannels { get; } |
|
int PatchSize { get; } |
Gets the patch size for latent tokenization. |
int TimeEmbeddingDim { get; } |
Methods
| Method | Summary |
|---|---|
Tensor<T> PredictNoise(Tensor<T> noisySample, int timestep, Tensor<T> conditioning) |
|
Tensor<T> PredictNoiseWithEmbedding(Tensor<T> noisySample, Tensor<T> timeEmbedding, Tensor<T> conditioning) |