GroupedQueryAttentionLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
Implements Grouped-Query Attention (GQA) from Ainslie et al., 2023.
Inheritance: LayerBase<T> → GroupedQueryAttentionLayer<T>
Implements: IShapeContract
For Beginners
GQA is a memory-efficient attention mechanism used by modern LLMs.
In standard attention with 64 heads:
- 64 Query projections, 64 Key projections, 64 Value projections
- KV-cache stores 64 sets of keys and values per layer
With GQA (64 Q heads, 8 KV heads):
- 64 Query projections, but only 8 Key and 8 Value projections
- Each K/V head is shared by 8 Query heads (64/8 = 8)
- KV-cache stores only 8 sets → 8x less memory!
Used by Llama 2 70B, Llama 3, Mistral, Gemma 2, and most modern large LLMs.
How It Works
GQA generalizes standard Multi-Head Attention (MHA) and Multi-Query Attention (MQA). Instead of having one set of Key/Value projections per head, GQA uses fewer K/V heads that are shared among groups of Query heads. This reduces the KV-cache memory by a factor of numHeads/numKVHeads while preserving most of the model quality.
When numKVHeads == numHeads, GQA is equivalent to standard MHA. When numKVHeads == 1, GQA is equivalent to MQA.
Constructors
| Constructor | Summary |
|---|---|
GroupedQueryAttentionLayer(int sequenceLength, int embeddingDimension, int numHeads, int numKVHeads, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy, bool deferAllocation, int? headDimension, bool useProjectionBias, double attnLogitSoftcap, bool useCausalMask) |
Creates a new Grouped-Query Attention layer. |
Properties
| Property | Summary |
|---|---|
AttentionVariant Variant { get; } |
Gets the attention variant this layer implements. |
PositionalEncodingType PositionalEncoding { get; set; } |
Gets the positional encoding type used by this attention layer. |
bool IsInitialized { get; } |
A deferred-allocation layer reports NOT initialized until its weights are materialized. |
bool SupportsTraining { get; } |
|
bool UsesProjectionBias { get; } |
Gets whether this layer adds separate learned biases to the Q/K/V projections (e.g. |
double AttnLogitSoftcap { get; } |
Gets the attention-logit soft-cap magnitude (Gemma-2 attn_logit_softcapping); 0 when disabled. |
double RoPETheta { get; } |
Gets the RoPE theta parameter if RoPE is configured, or the default 10000.0. |
int HeadDimension { get; } |
Gets the dimension of each attention head. |
int HeadsPerGroup { get; } |
Gets the number of query heads per KV head group. |
int NumHeads { get; } |
Gets the total number of query heads. |
int NumKVHeads { get; } |
Gets the number of key/value heads (fewer than query heads in GQA). |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
Tensor<T> GetKeyWeights() |
Gets the key projection weights for external use. |
Tensor<T> GetOutputWeights() |
Gets the output projection weights for external use. |
Tensor<T> GetQueryWeights() |
Gets the query projection weights for external use (e.g., quantization). |
Tensor<T> GetValueWeights() |
Gets the value projection weights for external use. |
void ClearGradients() |
|
void ConfigurePositionalEncoding(PositionalEncodingType encodingType, double ropeTheta, int maxSequenceLength) |
Configures positional encoding for this GQA layer. |
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |