FlashAttentionLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
A multi-head attention layer using the Flash Attention algorithm for memory-efficient computation.
Inheritance: LayerBase<T> → FlashAttentionLayer<T>
Implements: IShapeContract
For Beginners
This is like MultiHeadAttentionLayer but faster and more memory-efficient.
Flash Attention is a breakthrough algorithm that makes transformers much faster:
- Standard attention: O(N^2) memory, slow for long sequences
- Flash Attention: O(N) memory, 2-4x faster
Use this layer when:
- Training with long sequences (1024+ tokens)
- Training large models with limited GPU memory
- You need faster training/inference
The output is mathematically identical to standard attention - only the computation is different.
How It Works
FlashAttentionLayer provides the same functionality as MultiHeadAttentionLayer but uses the Flash Attention algorithm which is 2-4x faster and uses significantly less memory. It can be used as a drop-in replacement in transformer architectures.
Constructors
| Constructor | Summary |
|---|---|
FlashAttentionLayer(int sequenceLength, int embeddingDimension, int headCount, FlashAttentionConfig config, IActivationFunction<T> activationFunction) |
Creates a new Flash Attention layer with vector activation function. |
FlashAttentionLayer(int sequenceLength, int embeddingDimension, int headCount, FlashAttentionConfig config, IVectorActivationFunction<T> vectorActivationFunction) |
Creates a new Flash Attention layer with vector activation function. |
Properties
| Property | Summary |
|---|---|
FlashAttentionConfig Config { get; } |
Gets the Flash Attention configuration. |
PositionalEncodingType PositionalEncoding { get; set; } |
Gets the positional encoding type used by this attention layer. |
bool SupportsTraining { get; } |
Gets whether this layer supports training. |
double RoPETheta { get; } |
Gets the RoPE base frequency (theta) if RoPE is configured. |
int HeadCount { get; } |
Gets the number of attention heads. |
int HeadDimension { get; } |
Gets the dimension of each attention head. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetDiagnostics() |
Gets diagnostic information about the layer. |
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
Tensor<T> GetKeyWeights() |
Gets the key projection weights. |
Tensor<T> GetOutputWeights() |
Gets the output projection weights. |
Tensor<T> GetQueryWeights() |
Gets the query projection weights (for external access/debugging). |
Tensor<T> GetValueWeights() |
Gets the value projection weights. |
void ConfigurePositionalEncoding(PositionalEncodingType encodingType, double ropeTheta, int maxSequenceLength) |
Configures positional encoding for this Flash Attention layer. |
void ResetState() |
Resets the layer’s internal state. |
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |