DiffusionAttention<T>
Models & Types · AiDotNet.Diffusion.Attention
Memory-efficient attention layer for diffusion models using Flash Attention.
Inheritance: LayerBase<T> → DiffusionAttention<T>
Implements: IShapeContract
For Beginners
Attention in diffusion models is computationally expensive.
For a 512x512 image at 8x downsampling:
- Sequence length = (512/8)^2 = 4096 tokens
- Standard attention: 4096 x 4096 = 16 million attention weights!
This layer uses Flash Attention, which computes exact attention without materializing that matrix.
With residualGroupNorm the layer is a whole U-Net attention block (Ho et al. 2020):
GroupNorm, attention, then the input added back. Without it the layer is attention alone, for a caller
that adds its own normalization and residual.
Usage:
var attention = new DiffusionAttention<float>(
channels: 320,
numHeads: 8,
spatialSize: 64);
var input = Tensor<float>.CreateRandom(1, 320, 64, 64); // [batch, channels, h, w]
var output = attention.Forward(input);
How It Works
This attention layer runs Flash Attention at every sequence length, providing significant memory and performance benefits for high-resolution image generation.
Constructors
| Constructor | Summary |
|---|---|
DiffusionAttention(int channels, int numHeads, int spatialSize, bool useCausalMask, bool residualGroupNorm) |
Initializes a new diffusion attention layer. |
Properties
| Property | Summary |
|---|---|
bool FlashAttentionEnabled { get; } |
Gets whether Flash Attention is enabled. |
bool SupportsTraining { get; } |
Whether this layer supports training. |
int Channels { get; } |
Gets the number of channels. |
int NumHeads { get; } |
Gets the number of attention heads. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetDiagnostics() |
Gets diagnostic information about the layer. |
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
void ResetState() |
Resets the layer’s internal state. |
void SetTrainingMode(bool isTraining) |
Propagates eval/training mode to the nested Flash / standard attention sublayers. |
void UpdateParameters(T learningRate) |
Updates parameters using computed gradients. |