CachedGroupedQueryAttention<T>

Models & Types · AiDotNet.Inference

Grouped-Query Attention with KV-Cache support for efficient autoregressive inference.

Inheritance: LayerBase<T> → CachedGroupedQueryAttention<T>

Implements: IShapeContract

For Beginners

This is a fast version of GQA for text generation.

Memory savings compared to cached standard MHA:

  • Llama 2 70B: numHeads=64, numKVHeads=8 → 8x less KV-cache memory
  • The cache stores only 8 K/V sets instead of 64
  • During attention, each K/V set is shared by 8 query heads

This is critical for serving large models with limited GPU memory.

How It Works

This layer extends GQA with KV-cache support. The cache stores only numKVHeads sets of keys and values (not numHeads), reducing cache memory proportionally. During generation, K/V heads are expanded to match Q heads only for the attention computation.

Constructors

Constructor Summary
CachedGroupedQueryAttention(int sequenceLength, int embeddingDimension, int numHeads, int numKVHeads, bool useFlashAttention, int layerIndex, bool useCausalMask, IActivationFunction<T> activationFunction) Creates a new cached GQA layer.

Properties

Property Summary
KVCache<T> Cache { get; set; } Gets or sets the KV-Cache.
PositionalEncodingType PositionalEncoding { get; set; } Gets the positional encoding type.
bool InferenceMode { get; set; } Gets or sets whether the layer is in inference mode (uses cache).
bool SupportsTraining { get; }
int HeadCount { get; } Gets the number of query heads.
int HeadDimension { get; } Gets the dimension of each attention head.
int KVHeadCount { get; } Gets the number of KV heads (may be less than HeadCount).
int LayerIndex { get; set; } Gets or sets the layer index for cache indexing.

Methods

Method Summary
Dictionary<string, string> GetDiagnostics()
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
void ConfigurePositionalEncoding(PositionalEncodingType encodingType, double ropeTheta, int maxSequenceLength) Configures positional encoding.
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).