FlashAttentionConfig
Options & Configuration · AiDotNet.NeuralNetworks.Attention
Configuration options for Flash Attention algorithm.
For Beginners
Flash Attention is a faster way to compute attention.
Standard attention creates a huge matrix comparing every position to every other position. For long sequences (like 4096 tokens), this matrix has 16 million entries!
Flash Attention avoids creating this huge matrix by:
- Processing in small blocks that fit in fast GPU memory (SRAM)
- Computing softmax incrementally as it processes each block
- Never storing the full attention matrix
Benefits:
- 2-4x faster than standard attention
- Uses much less memory (O(N) instead of O(N^2))
- Enables training with longer sequences
How It Works
Flash Attention is a memory-efficient attention algorithm that avoids materializing the full N x N attention matrix. Instead, it processes attention in tiles/blocks, computing online softmax incrementally.
Constructors
| Constructor | Summary |
|---|---|
FlashAttentionConfig() |
Properties
| Property | Summary |
|---|---|
FlashAttentionConfig Causal { get; } |
Creates a configuration optimized for causal/autoregressive models. |
FlashAttentionConfig Default { get; } |
Creates a default configuration suitable for most use cases. |
FlashAttentionConfig HighPerformance { get; } |
Creates a configuration optimized for speed (uses more memory). |
FlashAttentionConfig MemoryEfficient { get; } |
Creates a configuration optimized for memory efficiency. |
FlashAttentionPrecision Precision { get; set; } |
Numerical precision mode for attention computation. |
bool RecomputeInBackward { get; set; } |
Whether to enable memory-efficient backward pass with recomputation. |
bool ReturnAttentionWeights { get; set; } |
Whether to return attention weights (for visualization/debugging). |
bool UseCausalMask { get; set; } |
Whether to apply causal masking (for autoregressive models). |
bool UseGpuKernel { get; set; } |
Whether to use the optimized GPU kernel (when available). |
float DropoutProbability { get; set; } |
Dropout probability to apply to attention weights during training. |
float? ScaleFactor { get; set; } |
Scale factor for attention scores. |
int BlockSizeKV { get; set; } |
Block size for key/value processing (Bc in the paper). |
int BlockSizeQ { get; set; } |
Block size for query processing (Br in the paper). |