CachedMultiHeadAttention<T>

Models & Types · AiDotNet.Inference

Multi-head attention layer with KV-Cache support for efficient autoregressive inference.

Inheritance: LayerBase<T> → CachedMultiHeadAttention<T>

Implements: IShapeContract

For Beginners

This is a fast version of attention for text generation.

Normal attention recalculates everything for each new token:

  • Token 1: Process token 1
  • Token 2: Process tokens 1-2 (redo token 1!)
  • Token 3: Process tokens 1-3 (redo tokens 1-2!)
  • … gets slower and slower

Cached attention remembers previous computations:

  • Token 1: Compute and cache K, V for token 1
  • Token 2: Only compute K, V for token 2, use cache for token 1
  • Token 3: Only compute K, V for token 3, use cache for tokens 1-2
  • … stays fast!

Use this layer when:

  • Generating text token by token (autoregressive)
  • Running inference (not training)
  • You want fast generation speed

How It Works

CachedMultiHeadAttention wraps standard multi-head attention with KV-Cache support. It automatically caches Key and Value projections across inference steps, enabling efficient token-by-token generation.

Constructors

Constructor Summary
CachedMultiHeadAttention(int sequenceLength, int embeddingDimension, int headCount, bool useFlashAttention, int layerIndex, bool useCausalMask, IActivationFunction<T> activationFunction) Creates a new cached multi-head attention layer.

Properties

Property Summary
KVCache<T> Cache { get; set; } Gets or sets the KV-Cache.
PositionalEncodingType PositionalEncoding { get; set; } Gets the positional encoding type used by this attention layer.
bool InferenceMode { get; set; } Gets or sets whether the layer is in inference mode (uses cache).
bool SupportsTraining { get; } Gets whether this layer supports training.
bool UsesCausalMask { get; } Gets whether causal masking is enabled for attention.
bool UsesFlashAttention { get; } Gets whether Flash Attention is enabled.
int HeadCount { get; } Gets the number of attention heads.
int HeadDimension { get; } Gets the dimension of each attention head.
int LayerIndex { get; set; } Gets or sets the layer index in the transformer (for cache indexing).

Methods

Method Summary
Dictionary<string, string> GetDiagnostics() Gets diagnostic information.
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
void ClearCache() Clears the KV-Cache if attached.
void ConfigurePositionalEncoding(PositionalEncodingType encodingType, double ropeTheta, int maxSequenceLength) Configures positional encoding for this cached attention layer.
void ResetState() Resets the layer’s state.
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate) Updates parameters using computed gradients.
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).