AttentionLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
Represents an Attention Layer for focusing on relevant parts of input sequences.
Inheritance: LayerBase<T> → AttentionLayer<T>
Implements: IAuxiliaryLossLayer<T>, IShapeContract
For Beginners
An Attention Layer helps the network focus on important parts of the input.
Think of it like reading a long document to answer a question:
- Instead of remembering every word, you focus on key sentences or phrases
- The attention mechanism does something similar for the neural network
- It helps the network decide which parts of the input are most relevant for the current task
Common applications include:
- Machine translation (focusing on relevant words when translating)
- Image captioning (focusing on relevant parts of an image when describing it)
- Speech recognition (focusing on important audio segments)
The key advantage is that it allows the network to handle long sequences more effectively by focusing on the most relevant parts rather than trying to remember everything.
How It Works
The Attention Layer is a mechanism that allows a neural network to focus on different parts of the input sequence when producing each element of the output sequence. It computes a weighted sum of the input sequence, where the weights (attention weights) are determined based on the relevance of each input element to the current output.
Constructors
| Constructor | Summary |
|---|---|
AttentionLayer(int attentionSize, IActivationFunction<T> activation, IInitializationStrategy<T> initializationStrategy) |
Initializes a new instance of the AttentionLayer class with scalar activation. |
AttentionLayer(int attentionSize, IVectorActivationFunction<T> activation) |
Initializes a new instance of the AttentionLayer class with vector activation. |
Properties
| Property | Summary |
|---|---|
AttentionKernelPath LastAttentionKernelPath { get; set; } |
Gets the attention kernel the most recent forward pass executed (#2312). |
IReadOnlyList<LayerPort> InputPorts { get; } |
Declares named input ports for this multi-input layer. |
T AuxiliaryLossWeight { get; set; } |
Gets or sets the weight for attention entropy regularization. |
bool SupportsTraining { get; } |
The computation engine (CPU or GPU) for vectorized operations. |
bool UseAuxiliaryLoss { get; set; } |
Gets or sets whether to use auxiliary loss (attention entropy regularization) during training. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetAuxiliaryLossDiagnostics() |
Gets diagnostic information about the attention regularization. |
Dictionary<string, string> GetDiagnostics() |
Gets diagnostic information about this component’s state and behavior. |
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
T ComputeAuxiliaryLoss() |
Computes the auxiliary loss for the AttentionLayer, which is attention entropy regularization. |
void ClearGradients() |
|
void ResetState() |
Resets the state of the attention layer. |
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void UpdateParameters(Vector<T> parameters) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |