CachedGroupedQueryAttention<T>
Models & Types · AiDotNet.Inference
Grouped-Query Attention with KV-Cache support for efficient autoregressive inference.
Inheritance: LayerBase<T> → CachedGroupedQueryAttention<T>
Implements: IShapeContract
For Beginners
This is a fast version of GQA for text generation.
Memory savings compared to cached standard MHA:
- Llama 2 70B: numHeads=64, numKVHeads=8 → 8x less KV-cache memory
- The cache stores only 8 K/V sets instead of 64
- During attention, each K/V set is shared by 8 query heads
This is critical for serving large models with limited GPU memory.
How It Works
This layer extends GQA with KV-cache support. The cache stores only numKVHeads sets of keys and values (not numHeads), reducing cache memory proportionally. During generation, K/V heads are expanded to match Q heads only for the attention computation.
Constructors
| Constructor | Summary |
|---|---|
CachedGroupedQueryAttention(int sequenceLength, int embeddingDimension, int numHeads, int numKVHeads, bool useFlashAttention, int layerIndex, bool useCausalMask, IActivationFunction<T> activationFunction) |
Creates a new cached GQA layer. |
Properties
| Property | Summary |
|---|---|
KVCache<T> Cache { get; set; } |
Gets or sets the KV-Cache. |
PositionalEncodingType PositionalEncoding { get; set; } |
Gets the positional encoding type. |
bool InferenceMode { get; set; } |
Gets or sets whether the layer is in inference mode (uses cache). |
bool SupportsTraining { get; } |
|
int HeadCount { get; } |
Gets the number of query heads. |
int HeadDimension { get; } |
Gets the dimension of each attention head. |
int KVHeadCount { get; } |
Gets the number of KV heads (may be less than HeadCount). |
int LayerIndex { get; set; } |
Gets or sets the layer index for cache indexing. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetDiagnostics() |
|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
void ConfigurePositionalEncoding(PositionalEncodingType encodingType, double ropeTheta, int maxSequenceLength) |
Configures positional encoding. |
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |