KVCache<T>

Models & Types · AiDotNet.Inference

Key-Value cache for efficient autoregressive inference in transformer models.

For Beginners

KV-Cache is like a memory bank for transformers.

When generating text:

  1. First token: Compute and cache K, V for position 0
  2. Second token: Compute K, V for position 1, append to cache, attend to positions 0-1
  3. Third token: Compute K, V for position 2, append to cache, attend to positions 0-2

… and so on

Without caching, token N would require recomputing K, V for positions 0 to N-1. With caching, we only compute K, V for the new position and look up the rest.

This provides massive speedup for autoregressive generation:

  • Without cache: O(N^2) total compute for N tokens
  • With cache: O(N) total compute for N tokens

How It Works

The KV-Cache stores computed Key and Value projections from attention layers, enabling efficient incremental generation where each new token only needs to compute attention against cached keys/values rather than recomputing everything.

Constructors

Constructor Summary
KVCache(KVCacheConfig config) Creates a new KV-Cache with the specified configuration.
KVCache(int numLayers, int numHeads, int headDim, int maxSeqLen, int maxBatchSize) Creates a new KV-Cache with default configuration.

Properties

Property Summary
KVCacheConfig Config { get; } Gets the configuration used for this cache.
int CurrentLength { get; } Gets the current number of cached tokens for batch item 0.
int MaxLength { get; } Gets the maximum sequence length this cache can hold.
long CacheHits { get; } Gets the number of cache hits (successful lookups).
long CacheMisses { get; } Gets the number of cache misses (new computations needed).
long Evictions { get; } Gets the number of evicted entries (due to sliding window).

Methods

Method Summary
Dictionary<string, object> GetStatistics() Gets cache statistics as a dictionary.
ValueTuple<Tensor<T>, Tensor<T>> Append(int layerIndex, Tensor<T> newKeys, Tensor<T> newValues) Appends new key-value pairs to the cache for a specific layer.
ValueTuple<Tensor<T>, Tensor<T>> GetCached(int layerIndex, int batchSize) Gets cached keys and values for a specific layer up to the current sequence length.
int GetSequenceLength(int batchIndex) Gets the current sequence length for a batch item.
long GetCurrentMemoryUsage() Gets the current memory usage of the cache in bytes.
void Clear() Clears all cached entries.
void Clear(int batchIndex) Clears cache for a specific batch index.
void CopyBatchState(int sourceBatch, int destBatch) Copies cache state from one batch index to another (for beam search).
void Truncate(int newLength, int batchIndex) Truncates the cache to a specific length (for beam search or rejection).
void Update(int layerIndex, int[] positions, Tensor<T> keys, Tensor<T> values) Updates cached keys and values at specific positions (for speculative decoding).