MultiHeadAttentionLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Implements a multi-head attention layer for neural networks, a key component in transformer architectures.

Inheritance: LayerBase<T> → MultiHeadAttentionLayer<T>

Implements: IAuxiliaryLossLayer<T>, IShapeContract

For Beginners

Multi-head attention is like having multiple “experts” look at the same information from different perspectives. Each “head” focuses on different parts of the input, allowing the model to capture various relationships in the data simultaneously. This is similar to how you might ask several friends for advice on a decision - each person might notice different important factors.

How It Works

Thread Safety: This layer is not thread-safe. Each layer instance maintains internal state during forward and backward passes. If you need concurrent execution, use separate layer instances per thread or synchronize access to shared instances.

Constructors

Constructor Summary
MultiHeadAttentionLayer(int headCount, int headDimension, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) Creates a new multi-head attention layer with the specified dimensions and head count.
MultiHeadAttentionLayer(int headCount, int headDimension, IVectorActivationFunction<T> vectorActivationFunction, IInitializationStrategy<T> initializationStrategy) Creates a new multi-head attention layer with the specified dimensions and head count.

Properties

Property Summary
AttentionKernelPath LastAttentionKernelPath { get; set; } Gets the attention kernel the most recent forward pass executed.
IQkvTransform<T> QkvTransform { get; set; } Optional hook that rewrites the projected queries, keys and values before they are split into heads.
IReadOnlyList<LayerPort> InputPorts { get; }
PositionalEncodingType PositionalEncoding { get; set; } Gets the positional encoding type used by this attention layer.
T AuxiliaryLossWeight { get; set; } Gets or sets the weight for the attention entropy auxiliary loss.
T HeadDiversityWeight { get; set; } Gets or sets the weight for head diversity penalty.
bool HasProjectionBiases { get; } Whether the query or value projection has a bias; see ConfigureProjectionBiases.
bool SupportsTraining { get; } The computation engine (CPU or GPU) for vectorized operations.
bool UseAuxiliaryLoss { get; set; } Gets or sets whether auxiliary loss (attention regularization) should be used during training.
bool UseCausalMask { get; set; } Gets or sets whether causal masking is applied during attention computation.
double AttentionDropoutRate { get; } Gets the dropout probability applied to the attention weights (the softmax output) during training.
double RoPETheta { get; } Gets the RoPE theta parameter if RoPE is configured, or the default 10000.0.
double RopeTheta { get; } RoPE base frequency (theta) last passed to ConfigurePositionalEncoding.
int HeadCount { get; } Gets the number of attention heads in this layer.
int HeadDimension { get; } Per-head embedding dimension (headCount × headDimension = embeddingDimension).
int PositionalMaxSequenceLength { get; } Maximum sequence length last passed to ConfigurePositionalEncoding.

Methods

Method Summary
Dictionary<string, string> GetAuxiliaryLossDiagnostics() Gets diagnostic information about the attention regularization auxiliary loss.
Dictionary<string, string> GetDiagnostics() Gets diagnostic information about this component’s state and behavior.
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
IReadOnlyList<ValueTuple<string, Tensor<T>>> GetRegisteredBuffers()
MultiHeadAttentionLayer<T> ConfigureAttentionDropout(double dropoutRate) Configures dropout on the attention probabilities (the softmax output), active in training mode only.
MultiHeadAttentionLayer<T> ConfigureAttentionGrouping(int groupSize)
MultiHeadAttentionLayer<T> ConfigureProjectionBiases(bool query, bool value) Configures projected-frame grouping for EfficientConformer attention.
T ComputeAuxiliaryLoss() Computes the auxiliary loss for attention regularization (entropy + head diversity).
Tensor<T> GetKeyWeights() Gets the key projection weights tensor for JIT compilation.
Tensor<T> GetOutputWeights() Gets the output projection weights tensor for JIT compilation.
Tensor<T> GetQueryWeights() Gets the query projection weights tensor for JIT compilation.
Tensor<T> GetValueWeights() Gets the value projection weights tensor for JIT compilation.
void ClearGradients()
void ConfigurePositionalEncoding(PositionalEncodingType encodingType, double ropeTheta, int maxSequenceLength) Configures positional encoding for this attention layer.
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void SetTrainingMode(bool isTraining) Overrides the base SetTrainingMode to drop training-only caches when transitioning to eval mode.
void UpdateParameters(T learningRate) Updates the layer’s parameters (weights and biases) using the calculated gradients.
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).