CachedMultiHeadAttention<T>
Models & Types · AiDotNet.Inference
Multi-head attention layer with KV-Cache support for efficient autoregressive inference.
Inheritance: LayerBase<T> → CachedMultiHeadAttention<T>
Implements: IShapeContract
For Beginners
This is a fast version of attention for text generation.
Normal attention recalculates everything for each new token:
- Token 1: Process token 1
- Token 2: Process tokens 1-2 (redo token 1!)
- Token 3: Process tokens 1-3 (redo tokens 1-2!)
- … gets slower and slower
Cached attention remembers previous computations:
- Token 1: Compute and cache K, V for token 1
- Token 2: Only compute K, V for token 2, use cache for token 1
- Token 3: Only compute K, V for token 3, use cache for tokens 1-2
- … stays fast!
Use this layer when:
- Generating text token by token (autoregressive)
- Running inference (not training)
- You want fast generation speed
How It Works
CachedMultiHeadAttention wraps standard multi-head attention with KV-Cache support. It automatically caches Key and Value projections across inference steps, enabling efficient token-by-token generation.
Constructors
| Constructor | Summary |
|---|---|
CachedMultiHeadAttention(int sequenceLength, int embeddingDimension, int headCount, bool useFlashAttention, int layerIndex, bool useCausalMask, IActivationFunction<T> activationFunction) |
Creates a new cached multi-head attention layer. |
Properties
| Property | Summary |
|---|---|
KVCache<T> Cache { get; set; } |
Gets or sets the KV-Cache. |
PositionalEncodingType PositionalEncoding { get; set; } |
Gets the positional encoding type used by this attention layer. |
bool InferenceMode { get; set; } |
Gets or sets whether the layer is in inference mode (uses cache). |
bool SupportsTraining { get; } |
Gets whether this layer supports training. |
bool UsesCausalMask { get; } |
Gets whether causal masking is enabled for attention. |
bool UsesFlashAttention { get; } |
Gets whether Flash Attention is enabled. |
int HeadCount { get; } |
Gets the number of attention heads. |
int HeadDimension { get; } |
Gets the dimension of each attention head. |
int LayerIndex { get; set; } |
Gets or sets the layer index in the transformer (for cache indexing). |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetDiagnostics() |
Gets diagnostic information. |
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
void ClearCache() |
Clears the KV-Cache if attached. |
void ConfigurePositionalEncoding(PositionalEncodingType encodingType, double ropeTheta, int maxSequenceLength) |
Configures positional encoding for this cached attention layer. |
void ResetState() |
Resets the layer’s state. |
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
Updates parameters using computed gradients. |
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |