PagedCachedMultiHeadAttention<T>
Models & Types · AiDotNet.Inference
Multi-head attention layer backed by PagedKVCache for efficient multi-sequence inference.
Inheritance: LayerBase<T> → PagedCachedMultiHeadAttention<T>
Implements: IShapeContract
How It Works
Limitation: This layer currently supports batchSize == 1 per sequence to avoid cache mixing.
For concurrent serving, create one sequence per request (distinct SequenceId values).
Constructors
| Constructor | Summary |
|---|---|
PagedCachedMultiHeadAttention(int sequenceLength, int embeddingDimension, int headCount, bool useCausalMask, IActivationFunction<T> activationFunction, int kvHeadCount) |
Properties
| Property | Summary |
|---|---|
PagedAttentionKernel<T> Kernel { get; set; } |
Gets or sets the PagedAttention kernel (owns the paged cache). |
PositionalEncodingType PositionalEncoding { get; set; } |
Gets the positional encoding type used by this attention layer. |
bool InferenceMode { get; set; } |
Gets or sets whether the layer is in inference mode (uses paged cache). |
bool SupportsTraining { get; } |
Gets whether this layer supports training. |
double RoPETheta { get; } |
Gets the RoPE base frequency (theta) if RoPE is configured. |
int HeadCount { get; } |
Gets the number of query attention heads. |
int HeadDimension { get; } |
Gets the dimension of each attention head. |
int KVHeadCount { get; } |
Gets the number of key/value attention heads. |
int LayerIndex { get; set; } |
Gets or sets the layer index for KV-cache addressing. |
long SequenceId { get; set; } |
Gets or sets the sequence ID used for this layer’s cache operations. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
Tensor<T> ForwardWithContext(Tensor<T> input, InferenceForwardContext ctx) |
|
void ConfigurePositionalEncoding(PositionalEncodingType encodingType, double ropeTheta, int maxSequenceLength) |
Configures positional encoding for this paged cached attention layer. |
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |