KVCacheConfig

Options & Configuration · AiDotNet.Inference

Configuration for Key-Value cache used in autoregressive inference.

For Beginners

KV-Cache makes text generation much faster.

When generating text token by token:

  • Without cache: Generate token 100 by processing tokens 1-99 again (slow!)
  • With cache: Generate token 100 using cached computations from tokens 1-99 (fast!)

This can speed up generation by 10-100x for long sequences.

The cache stores the Key and Value projections from attention layers, which don’t change once computed for a given position.

How It Works

KV-Cache is essential for efficient autoregressive generation (like in GPT models). Without caching, each new token requires recomputing attention for all previous tokens. With caching, we only compute attention for the new token and look up cached keys/values.

Constructors

Constructor Summary
KVCacheConfig()

Properties

Property Summary
CacheDataType DataType { get; set; } Data type for cache storage.
CacheDevice Device { get; set; } Device placement for the cache (CPU or GPU).
bool PreAllocate { get; set; } Whether to pre-allocate all memory at initialization.
bool UseSlidingWindow { get; set; } Whether to use sliding window attention (for long sequences).
int HeadDimension { get; set; } Dimension of each attention head.
int MaxBatchSize { get; set; } Maximum batch size for the cache.
int MaxSequenceLength { get; set; } Maximum sequence length the cache can hold.
int NumHeads { get; set; } Number of attention heads per layer.
int NumLayers { get; set; } Number of transformer layers to cache.
int WindowSize { get; set; } Size of sliding window (if enabled).

Methods

Method Summary
KVCacheConfig ForModel(string modelSize) Creates a default configuration for common model sizes.
long EstimateMemoryBytes() Computes the total memory required for the cache in bytes.