PagedKVCacheConfig

Options & Configuration · AiDotNet.Inference.PagedAttention

Configuration for PagedKVCache.

Constructors

Constructor Summary
PagedKVCacheConfig()

Properties

Property Summary
int BlockSize { get; set; } Number of tokens per block.
int HeadDimension { get; set; } Dimension of each head.
int MaxSeqLen { get; set; } Maximum sequence length (for efficiency calculation).
int NumBlocks { get; set; } Total number of blocks.
int NumHeads { get; set; } Number of attention heads.
int NumLayers { get; set; } Number of transformer layers.

Methods

Method Summary
PagedKVCacheConfig ForModel(string modelName, long availableBytes, int blockSize) Creates configuration for a specific model.
PagedKVCacheConfig FromMemorySize(long availableBytes, int numLayers, int numHeads, int headDim, int blockSize) Creates configuration from available memory.