FlashAttentionLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

A multi-head attention layer using the Flash Attention algorithm for memory-efficient computation.

Inheritance: LayerBase<T> → FlashAttentionLayer<T>

Implements: IShapeContract

For Beginners

This is like MultiHeadAttentionLayer but faster and more memory-efficient.

Flash Attention is a breakthrough algorithm that makes transformers much faster:

  • Standard attention: O(N^2) memory, slow for long sequences
  • Flash Attention: O(N) memory, 2-4x faster

Use this layer when:

  • Training with long sequences (1024+ tokens)
  • Training large models with limited GPU memory
  • You need faster training/inference

The output is mathematically identical to standard attention - only the computation is different.

How It Works

FlashAttentionLayer provides the same functionality as MultiHeadAttentionLayer but uses the Flash Attention algorithm which is 2-4x faster and uses significantly less memory. It can be used as a drop-in replacement in transformer architectures.

Constructors

Constructor Summary
FlashAttentionLayer(int sequenceLength, int embeddingDimension, int headCount, FlashAttentionConfig config, IActivationFunction<T> activationFunction) Creates a new Flash Attention layer with vector activation function.
FlashAttentionLayer(int sequenceLength, int embeddingDimension, int headCount, FlashAttentionConfig config, IVectorActivationFunction<T> vectorActivationFunction) Creates a new Flash Attention layer with vector activation function.

Properties

Property Summary
FlashAttentionConfig Config { get; } Gets the Flash Attention configuration.
PositionalEncodingType PositionalEncoding { get; set; } Gets the positional encoding type used by this attention layer.
bool SupportsTraining { get; } Gets whether this layer supports training.
double RoPETheta { get; } Gets the RoPE base frequency (theta) if RoPE is configured.
int HeadCount { get; } Gets the number of attention heads.
int HeadDimension { get; } Gets the dimension of each attention head.

Methods

Method Summary
Dictionary<string, string> GetDiagnostics() Gets diagnostic information about the layer.
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
Tensor<T> GetKeyWeights() Gets the key projection weights.
Tensor<T> GetOutputWeights() Gets the output projection weights.
Tensor<T> GetQueryWeights() Gets the query projection weights (for external access/debugging).
Tensor<T> GetValueWeights() Gets the value projection weights.
void ConfigurePositionalEncoding(PositionalEncodingType encodingType, double ropeTheta, int maxSequenceLength) Configures positional encoding for this Flash Attention layer.
void ResetState() Resets the layer’s internal state.
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).