GatedLinearAttentionLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers.SSM
Implements the Gated Linear Attention (GLA) layer from Yang et al., 2023.
Inheritance: LayerBase<T> → GatedLinearAttentionLayer<T>
Implements: IShapeContract
For Beginners
GLA is a way to make attention fast by adding a “memory gate.”
Standard attention: “Look at everything” -> O(n^2) cost Linear attention: “Keep a running summary” -> O(n) cost, but forgets old info uniformly GLA: “Keep a running summary with a gate that controls forgetting” -> O(n) cost, smart forgetting
The gate lets the model decide: “This information is important, keep it longer” or “This is no longer relevant, forget it faster.” This simple addition makes linear attention much more competitive with standard attention.
How It Works
GLA materializes the linear attention mechanism as a gated linear RNN with data-dependent gating. It bridges the gap between linear attention (which can be computed as an RNN) and traditional gated RNNs (which have data-dependent transitions), resulting in a model that is both expressive and hardware-efficient.
The computation:
The key insight is that the gate G controls how quickly the state forgets old information, making the effective attention window data-dependent.
GLA supports a hardware-efficient chunked computation mode where the sequence is processed in blocks, with intra-chunk computation parallelized as matrix multiplications.
Reference: Yang et al., “Gated Linear Attention Transformers with Hardware-Efficient Training”, 2024.
https://arxiv.org/abs/2312.06635
Constructors
| Constructor | Summary |
|---|---|
GatedLinearAttentionLayer(int sequenceLength, int modelDimension, int numHeads, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy) |
Creates a new Gated Linear Attention layer. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int HeadDimension { get; } |
Gets the dimension per head. |
int ModelDimension { get; } |
Gets the model dimension. |
int NumHeads { get; } |
Gets the number of attention heads. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
Tensor<T> GetOutputWeights() |
Gets a copy of the output projection weights for external inspection. |
Tensor<T> GetQueryWeights() |
Gets a copy of the query projection weights for external inspection. |
void ClearGradients() |
|
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |