GroupedQueryAttentionLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Implements Grouped-Query Attention (GQA) from Ainslie et al., 2023.

Inheritance: LayerBase<T> → GroupedQueryAttentionLayer<T>

Implements: IShapeContract

For Beginners

GQA is a memory-efficient attention mechanism used by modern LLMs.

In standard attention with 64 heads:

  • 64 Query projections, 64 Key projections, 64 Value projections
  • KV-cache stores 64 sets of keys and values per layer

With GQA (64 Q heads, 8 KV heads):

  • 64 Query projections, but only 8 Key and 8 Value projections
  • Each K/V head is shared by 8 Query heads (64/8 = 8)
  • KV-cache stores only 8 sets → 8x less memory!

Used by Llama 2 70B, Llama 3, Mistral, Gemma 2, and most modern large LLMs.

How It Works

GQA generalizes standard Multi-Head Attention (MHA) and Multi-Query Attention (MQA). Instead of having one set of Key/Value projections per head, GQA uses fewer K/V heads that are shared among groups of Query heads. This reduces the KV-cache memory by a factor of numHeads/numKVHeads while preserving most of the model quality.

When numKVHeads == numHeads, GQA is equivalent to standard MHA. When numKVHeads == 1, GQA is equivalent to MQA.

Constructors

Constructor Summary
GroupedQueryAttentionLayer(int sequenceLength, int embeddingDimension, int numHeads, int numKVHeads, IActivationFunction<T> activationFunction, IInitializationStrategy<T> initializationStrategy, bool deferAllocation, int? headDimension, bool useProjectionBias, double attnLogitSoftcap, bool useCausalMask) Creates a new Grouped-Query Attention layer.

Properties

Property Summary
AttentionVariant Variant { get; } Gets the attention variant this layer implements.
PositionalEncodingType PositionalEncoding { get; set; } Gets the positional encoding type used by this attention layer.
bool IsInitialized { get; } A deferred-allocation layer reports NOT initialized until its weights are materialized.
bool SupportsTraining { get; }
bool UsesProjectionBias { get; } Gets whether this layer adds separate learned biases to the Q/K/V projections (e.g.
double AttnLogitSoftcap { get; } Gets the attention-logit soft-cap magnitude (Gemma-2 attn_logit_softcapping); 0 when disabled.
double RoPETheta { get; } Gets the RoPE theta parameter if RoPE is configured, or the default 10000.0.
int HeadDimension { get; } Gets the dimension of each attention head.
int HeadsPerGroup { get; } Gets the number of query heads per KV head group.
int NumHeads { get; } Gets the total number of query heads.
int NumKVHeads { get; } Gets the number of key/value heads (fewer than query heads in GQA).

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
Tensor<T> GetKeyWeights() Gets the key projection weights for external use.
Tensor<T> GetOutputWeights() Gets the output projection weights for external use.
Tensor<T> GetQueryWeights() Gets the query projection weights for external use (e.g., quantization).
Tensor<T> GetValueWeights() Gets the value projection weights for external use.
void ClearGradients()
void ConfigurePositionalEncoding(PositionalEncodingType encodingType, double ropeTheta, int maxSequenceLength) Configures positional encoding for this GQA layer.
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).