DiffusionAttention<T>

Models & Types · AiDotNet.Diffusion.Attention

Memory-efficient attention layer for diffusion models using Flash Attention.

Inheritance: LayerBase<T> → DiffusionAttention<T>

Implements: IShapeContract

For Beginners

Attention in diffusion models is computationally expensive.

For a 512x512 image at 8x downsampling:

  • Sequence length = (512/8)^2 = 4096 tokens
  • Standard attention: 4096 x 4096 = 16 million attention weights!

This layer uses Flash Attention, which computes exact attention without materializing that matrix.

With residualGroupNorm the layer is a whole U-Net attention block (Ho et al. 2020): GroupNorm, attention, then the input added back. Without it the layer is attention alone, for a caller that adds its own normalization and residual.

Usage:

var attention = new DiffusionAttention<float>(
channels: 320,
numHeads: 8,
spatialSize: 64);

var input = Tensor<float>.CreateRandom(1, 320, 64, 64); // [batch, channels, h, w]
var output = attention.Forward(input);

How It Works

This attention layer runs Flash Attention at every sequence length, providing significant memory and performance benefits for high-resolution image generation.

Constructors

Constructor Summary
DiffusionAttention(int channels, int numHeads, int spatialSize, bool useCausalMask, bool residualGroupNorm) Initializes a new diffusion attention layer.

Properties

Property Summary
bool FlashAttentionEnabled { get; } Gets whether Flash Attention is enabled.
bool SupportsTraining { get; } Whether this layer supports training.
int Channels { get; } Gets the number of channels.
int NumHeads { get; } Gets the number of attention heads.

Methods

Method Summary
Dictionary<string, string> GetDiagnostics() Gets diagnostic information about the layer.
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
void ResetState() Resets the layer’s internal state.
void SetTrainingMode(bool isTraining) Propagates eval/training mode to the nested Flash / standard attention sublayers.
void UpdateParameters(T learningRate) Updates parameters using computed gradients.