MMDiTNoisePredictor<T>

Models & Types · AiDotNet.Diffusion.NoisePredictors

Multi-Modal Diffusion Transformer (MMDiT) noise predictor for SD3 and FLUX architectures.

Inheritance: NoisePredictorBase<T> → MMDiTNoisePredictor<T>

Known derived types: AsymmDiTPredictor<T>, EMMDiTPredictor<T>, FluxDoubleStreamPredictor<T>, MMDiTXNoisePredictor<T>

For Beginners

MMDiT is the architecture behind SD3 and FLUX:

How MMDiT differs from standard DiT:

  • Standard DiT: Image patches processed by transformer, text injected via cross-attention
  • MMDiT: Image AND text tokens are concatenated and processed together through JOINT attention

Key characteristics:

  • Joint attention: Both text and image tokens attend to each other equally
  • Separate MLPs: Text and image tokens have independent MLP layers after joint attention
  • Dual stream: Text and image streams with shared attention but separate feed-forward
  • AdaLN-Zero: Adaptive layer normalization with zero-init gating
  • Supports multiple text encoders (CLIP + T5 for SD3, CLIP + T5 for FLUX)

Used in:

  • Stable Diffusion 3 / SD 3.5 (Stability AI)
  • FLUX.1 dev/schnell/pro (Black Forest Labs)
  • Pixart-Sigma

Advantages over standard DiT:

  • Better text-image alignment through joint attention
  • More expressive conditioning without cross-attention bottleneck
  • Scales better with model size

How It Works

MMDiT extends the standard DiT architecture by processing text and image tokens jointly through shared transformer blocks, rather than using cross-attention. This enables deeper bidirectional interaction between modalities.

Technical specifications:

  • Architecture: Multi-modal transformer with joint self-attention
  • SD3 Medium: 2B params, 24 layers, hidden 1536, 24 heads
  • FLUX.1 dev: ~12B params, 19 double + 38 single layers, hidden 3072, 24 heads
  • Patch size: 2 (latent space)
  • Conditioning: Concatenated CLIP + T5 text embeddings
  • Timestep: Sinusoidal + MLP projection
  • Positional encoding: RoPE (Rotary Position Embedding)

Reference: Esser et al., “Scaling Rectified Flow Transformers for High-Resolution Image Synthesis”, ICML 2024

Example

// Create MMDiT for SD3
var mmdit = new MMDiTNoisePredictor<float>(
    inputChannels: 16,       // SD3 uses 16 latent channels
    hiddenSize: 1536,        // SD3 Medium
    numJointLayers: 24,
    numHeads: 24,
    contextDim: 4096);       // T5-XXL
            
// Create MMDiT for FLUX
var mmdit2 = new MMDiTNoisePredictor<float>(
    inputChannels: 16,
    hiddenSize: 3072,        // FLUX.1
    numJointLayers: 19,
    numSingleLayers: 38,
    numHeads: 24,
    contextDim: 4096);

Constructors

Constructor Summary
MMDiTNoisePredictor(NeuralNetworkArchitecture<T> architecture, int inputChannels, int hiddenSize, int numJointLayers, int numSingleLayers, int numHeads, int patchSize, int contextDim, double mlpRatio, List<MMDiTBlock<T>> customJointBlocks, List<MMDiTSingleBlock<T>> customSingleBlocks, ILossFunction<T> lossFunction, int? seed) Initializes a new instance of the MMDiTNoisePredictor class with full customization support.

Properties

Property Summary
bool SupportsCFG { get; }
bool SupportsCrossAttention { get; }
int BaseChannels { get; }
int ContextDimension { get; }
int HiddenSize { get; } Gets the hidden size of the transformer.
int InputChannels { get; }
int NumJointLayers { get; } Gets the number of joint (double-stream) transformer layers.
int NumSingleLayers { get; } Gets the number of single-stream transformer layers (FLUX-style).
int OutputChannels { get; }
int PatchSize { get; } Gets the patch size for latent tokenization.
int TimeEmbeddingDim { get; }

Methods

Method Summary
Tensor<T> PredictNoise(Tensor<T> noisySample, int timestep, Tensor<T> conditioning)
Tensor<T> PredictNoiseWithEmbedding(Tensor<T> noisySample, Tensor<T> timeEmbedding, Tensor<T> conditioning)