FlashAttentionConfig

Options & Configuration · AiDotNet.NeuralNetworks.Attention

Configuration options for Flash Attention algorithm.

For Beginners

Flash Attention is a faster way to compute attention.

Standard attention creates a huge matrix comparing every position to every other position. For long sequences (like 4096 tokens), this matrix has 16 million entries!

Flash Attention avoids creating this huge matrix by:

  • Processing in small blocks that fit in fast GPU memory (SRAM)
  • Computing softmax incrementally as it processes each block
  • Never storing the full attention matrix

Benefits:

  • 2-4x faster than standard attention
  • Uses much less memory (O(N) instead of O(N^2))
  • Enables training with longer sequences

How It Works

Flash Attention is a memory-efficient attention algorithm that avoids materializing the full N x N attention matrix. Instead, it processes attention in tiles/blocks, computing online softmax incrementally.

Constructors

Constructor Summary
FlashAttentionConfig()

Properties

Property Summary
FlashAttentionConfig Causal { get; } Creates a configuration optimized for causal/autoregressive models.
FlashAttentionConfig Default { get; } Creates a default configuration suitable for most use cases.
FlashAttentionConfig HighPerformance { get; } Creates a configuration optimized for speed (uses more memory).
FlashAttentionConfig MemoryEfficient { get; } Creates a configuration optimized for memory efficiency.
FlashAttentionPrecision Precision { get; set; } Numerical precision mode for attention computation.
bool RecomputeInBackward { get; set; } Whether to enable memory-efficient backward pass with recomputation.
bool ReturnAttentionWeights { get; set; } Whether to return attention weights (for visualization/debugging).
bool UseCausalMask { get; set; } Whether to apply causal masking (for autoregressive models).
bool UseGpuKernel { get; set; } Whether to use the optimized GPU kernel (when available).
float DropoutProbability { get; set; } Dropout probability to apply to attention weights during training.
float? ScaleFactor { get; set; } Scale factor for attention scores.
int BlockSizeKV { get; set; } Block size for key/value processing (Bc in the paper).
int BlockSizeQ { get; set; } Block size for query processing (Br in the paper).