PagedKVCacheConfig
Options & Configuration · AiDotNet.Inference.PagedAttention
Configuration for PagedKVCache.
Constructors
| Constructor |
Summary |
PagedKVCacheConfig() |
|
Properties
| Property |
Summary |
int BlockSize { get; set; } |
Number of tokens per block. |
int HeadDimension { get; set; } |
Dimension of each head. |
int MaxSeqLen { get; set; } |
Maximum sequence length (for efficiency calculation). |
int NumBlocks { get; set; } |
Total number of blocks. |
int NumHeads { get; set; } |
Number of attention heads. |
int NumLayers { get; set; } |
Number of transformer layers. |
Methods
| Method |
Summary |
PagedKVCacheConfig ForModel(string modelName, long availableBytes, int blockSize) |
Creates configuration for a specific model. |
PagedKVCacheConfig FromMemorySize(long availableBytes, int numLayers, int numHeads, int headDim, int blockSize) |
Creates configuration from available memory. |