KVCache<T>
Models & Types · AiDotNet.Inference
Key-Value cache for efficient autoregressive inference in transformer models.
For Beginners
KV-Cache is like a memory bank for transformers.
When generating text:
- First token: Compute and cache K, V for position 0
- Second token: Compute K, V for position 1, append to cache, attend to positions 0-1
- Third token: Compute K, V for position 2, append to cache, attend to positions 0-2
… and so on
Without caching, token N would require recomputing K, V for positions 0 to N-1. With caching, we only compute K, V for the new position and look up the rest.
This provides massive speedup for autoregressive generation:
- Without cache: O(N^2) total compute for N tokens
- With cache: O(N) total compute for N tokens
How It Works
The KV-Cache stores computed Key and Value projections from attention layers, enabling efficient incremental generation where each new token only needs to compute attention against cached keys/values rather than recomputing everything.
Constructors
| Constructor | Summary |
|---|---|
KVCache(KVCacheConfig config) |
Creates a new KV-Cache with the specified configuration. |
KVCache(int numLayers, int numHeads, int headDim, int maxSeqLen, int maxBatchSize) |
Creates a new KV-Cache with default configuration. |
Properties
| Property | Summary |
|---|---|
KVCacheConfig Config { get; } |
Gets the configuration used for this cache. |
int CurrentLength { get; } |
Gets the current number of cached tokens for batch item 0. |
int MaxLength { get; } |
Gets the maximum sequence length this cache can hold. |
long CacheHits { get; } |
Gets the number of cache hits (successful lookups). |
long CacheMisses { get; } |
Gets the number of cache misses (new computations needed). |
long Evictions { get; } |
Gets the number of evicted entries (due to sliding window). |
Methods
| Method | Summary |
|---|---|
Dictionary<string, object> GetStatistics() |
Gets cache statistics as a dictionary. |
ValueTuple<Tensor<T>, Tensor<T>> Append(int layerIndex, Tensor<T> newKeys, Tensor<T> newValues) |
Appends new key-value pairs to the cache for a specific layer. |
ValueTuple<Tensor<T>, Tensor<T>> GetCached(int layerIndex, int batchSize) |
Gets cached keys and values for a specific layer up to the current sequence length. |
int GetSequenceLength(int batchIndex) |
Gets the current sequence length for a batch item. |
long GetCurrentMemoryUsage() |
Gets the current memory usage of the cache in bytes. |
void Clear() |
Clears all cached entries. |
void Clear(int batchIndex) |
Clears cache for a specific batch index. |
void CopyBatchState(int sourceBatch, int destBatch) |
Copies cache state from one batch index to another (for beam search). |
void Truncate(int newLength, int batchIndex) |
Truncates the cache to a specific length (for beam search or rejection). |
void Update(int layerIndex, int[] positions, Tensor<T> keys, Tensor<T> values) |
Updates cached keys and values at specific positions (for speculative decoding). |