SelfAttentionLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Represents a self-attention layer that allows a sequence to attend to itself, capturing relationships between elements.

Inheritance: LayerBase<T> → SelfAttentionLayer<T>

Implements: IAuxiliaryLossLayer<T>, IShapeContract

For Beginners

This layer helps a neural network understand relationships between different parts of a sequence.

Think of the SelfAttentionLayer like a group of spotlights at a theater performance:

  • Each spotlight (attention head) can focus on different actors on stage
  • For each actor, the spotlights decide which other actors are most relevant to them
  • The spotlights assign importance scores to these relationships
  • This helps the network understand who is interacting with whom, and how

For example, in a sentence like “The cat sat on the mat because it was tired”:

  • Traditional networks might struggle to figure out what “it” refers to
  • Self-attention can learn that “it” has a strong relationship with “cat”
  • This helps the network understand that the cat was tired, not the mat

Multi-head attention (using multiple “spotlights”) allows the layer to focus on different types of relationships simultaneously, such as grammatical structure, semantic meaning, and contextual clues.

Self-attention is a cornerstone of modern natural language processing and has revolutionized how neural networks handle sequential data like text, time series, and even images.

How It Works

The SelfAttentionLayer implements the self-attention mechanism, a key component of transformer architectures. It allows each position in a sequence to attend to all positions within the same sequence, enabling the model to capture long-range dependencies and relationships. The layer uses the scaled dot-product attention mechanism with multiple attention heads, which allows it to focus on different aspects of the input simultaneously.

Constructors

Constructor Summary
SelfAttentionLayer(int sequenceLength, int embeddingDimension, int headCount, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) Initializes a new instance of the SelfAttentionLayer class with a vector activation function.
SelfAttentionLayer(int sequenceLength, int embeddingDimension, int headCount, IVectorActivationFunction<T> vectorActivationFunction, IInitializationStrategy<T> initializationStrategy) Initializes a new instance of the SelfAttentionLayer class with a vector activation function.

Properties

Property Summary
T AuxiliaryLossWeight { get; set; } Gets or sets the weight for the attention sparsity auxiliary loss.
bool SupportsTraining { get; } Gets a value indicating whether this layer supports training.
bool UseAuxiliaryLoss { get; set; } Gets or sets whether auxiliary loss (attention sparsity regularization) should be used during training.

Methods

Method Summary
Dictionary<string, string> GetAuxiliaryLossDiagnostics() Gets diagnostic information about the attention sparsity auxiliary loss.
Dictionary<string, string> GetDiagnostics() Gets diagnostic information about this component’s state and behavior.
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
IReadOnlyList<ValueTuple<string, Tensor<T>>> GetRegisteredBuffers()
T ComputeAuxiliaryLoss() Initializes the layer’s internal parameters based on the sequence length, embedding dimension, and head count.
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate) Updates the parameters of the self-attention layer using the calculated gradients.
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).