PagedCachedMultiHeadAttention<T>

Models & Types · AiDotNet.Inference

Multi-head attention layer backed by PagedKVCache for efficient multi-sequence inference.

Inheritance: LayerBase<T> → PagedCachedMultiHeadAttention<T>

Implements: IShapeContract

How It Works

Limitation: This layer currently supports batchSize == 1 per sequence to avoid cache mixing. For concurrent serving, create one sequence per request (distinct SequenceId values).

Constructors

Constructor Summary
PagedCachedMultiHeadAttention(int sequenceLength, int embeddingDimension, int headCount, bool useCausalMask, IActivationFunction<T> activationFunction, int kvHeadCount)

Properties

Property Summary
PagedAttentionKernel<T> Kernel { get; set; } Gets or sets the PagedAttention kernel (owns the paged cache).
PositionalEncodingType PositionalEncoding { get; set; } Gets the positional encoding type used by this attention layer.
bool InferenceMode { get; set; } Gets or sets whether the layer is in inference mode (uses paged cache).
bool SupportsTraining { get; } Gets whether this layer supports training.
double RoPETheta { get; } Gets the RoPE base frequency (theta) if RoPE is configured.
int HeadCount { get; } Gets the number of query attention heads.
int HeadDimension { get; } Gets the dimension of each attention head.
int KVHeadCount { get; } Gets the number of key/value attention heads.
int LayerIndex { get; set; } Gets or sets the layer index for KV-cache addressing.
long SequenceId { get; set; } Gets or sets the sequence ID used for this layer’s cache operations.

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
Tensor<T> ForwardWithContext(Tensor<T> input, InferenceForwardContext ctx)
void ConfigurePositionalEncoding(PositionalEncodingType encodingType, double ropeTheta, int maxSequenceLength) Configures positional encoding for this paged cached attention layer.
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).