SelfAttentionLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
Represents a self-attention layer that allows a sequence to attend to itself, capturing relationships between elements.
Inheritance: LayerBase<T> → SelfAttentionLayer<T>
Implements: IAuxiliaryLossLayer<T>, IShapeContract
For Beginners
This layer helps a neural network understand relationships between different parts of a sequence.
Think of the SelfAttentionLayer like a group of spotlights at a theater performance:
- Each spotlight (attention head) can focus on different actors on stage
- For each actor, the spotlights decide which other actors are most relevant to them
- The spotlights assign importance scores to these relationships
- This helps the network understand who is interacting with whom, and how
For example, in a sentence like “The cat sat on the mat because it was tired”:
- Traditional networks might struggle to figure out what “it” refers to
- Self-attention can learn that “it” has a strong relationship with “cat”
- This helps the network understand that the cat was tired, not the mat
Multi-head attention (using multiple “spotlights”) allows the layer to focus on different types of relationships simultaneously, such as grammatical structure, semantic meaning, and contextual clues.
Self-attention is a cornerstone of modern natural language processing and has revolutionized how neural networks handle sequential data like text, time series, and even images.
How It Works
The SelfAttentionLayer implements the self-attention mechanism, a key component of transformer architectures. It allows each position in a sequence to attend to all positions within the same sequence, enabling the model to capture long-range dependencies and relationships. The layer uses the scaled dot-product attention mechanism with multiple attention heads, which allows it to focus on different aspects of the input simultaneously.
Constructors
| Constructor | Summary |
|---|---|
SelfAttentionLayer(int sequenceLength, int embeddingDimension, int headCount, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) |
Initializes a new instance of the SelfAttentionLayer class with a vector activation function. |
SelfAttentionLayer(int sequenceLength, int embeddingDimension, int headCount, IVectorActivationFunction<T> vectorActivationFunction, IInitializationStrategy<T> initializationStrategy) |
Initializes a new instance of the SelfAttentionLayer class with a vector activation function. |
Properties
| Property | Summary |
|---|---|
T AuxiliaryLossWeight { get; set; } |
Gets or sets the weight for the attention sparsity auxiliary loss. |
bool SupportsTraining { get; } |
Gets a value indicating whether this layer supports training. |
bool UseAuxiliaryLoss { get; set; } |
Gets or sets whether auxiliary loss (attention sparsity regularization) should be used during training. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetAuxiliaryLossDiagnostics() |
Gets diagnostic information about the attention sparsity auxiliary loss. |
Dictionary<string, string> GetDiagnostics() |
Gets diagnostic information about this component’s state and behavior. |
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
IReadOnlyList<ValueTuple<string, Tensor<T>>> GetRegisteredBuffers() |
|
T ComputeAuxiliaryLoss() |
Initializes the layer’s internal parameters based on the sequence length, embedding dimension, and head count. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
Updates the parameters of the self-attention layer using the calculated gradients. |
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |