PagedKVCache<T>

Models & Types · AiDotNet.Inference.PagedAttention

Paged key-value cache for efficient LLM serving memory management.

Implements: IDisposable

For Beginners

Traditional KV-cache is like reserving a whole hotel floor per guest.

PagedKVCache is like renting hotel rooms individually:

  • Guest arrives: Get them a room (allocate 1 block)
  • Guest needs more space: Give them another room (allocate more blocks)
  • Guest leaves: Rooms become available (free blocks)

Benefits:

  • 8-9x more sequences can fit in memory
  • No wasted space for short sequences
  • Efficient beam search with copy-on-write

How It Works

PagedKVCache implements the vLLM-style paged attention memory management system. Instead of pre-allocating contiguous memory for each sequence’s maximum length, it dynamically allocates fixed-size blocks as sequences grow.

Constructors

Constructor Summary
PagedKVCache(PagedKVCacheConfig config) Creates a new paged KV cache.

Properties

Property Summary
BlockManager<T> BlockManager { get; } Gets the block manager.
BlockTableManager<T> BlockTableManager { get; } Gets the block table manager.
PagedKVCacheConfig Config { get; } Gets the configuration.
int ActiveSequenceCount { get; } Gets the number of active sequences.

Methods

Method Summary
IReadOnlyList<T[]> GetStorageBlocks() Gets the underlying storage blocks (for GPU transfer).
PagedKVCache<T> FromMemorySize(long availableBytes, int numLayers, int numHeads, int headDim, int blockSize) Creates a new paged KV cache from memory size.
PagedKVCacheStats GetStats() Gets statistics about the cache.
bool AllocateSequence(long sequenceId, int initialTokens) Allocates cache space for a new sequence.
bool ExtendSequence(long sequenceId, int additionalTokens) Extends a sequence’s cache for additional tokens.
bool ForkSequence(long sourceSequenceId, long newSequenceId) Forks a sequence’s cache for beam search.
bool HasCapacityFor(long sequenceId, int additionalTokens) Checks if more tokens can be added to a sequence without new allocation.
bool TruncateSequence(long sequenceId, int newLength) Rolls a sequence’s logical length back to newLength, discarding the KV at positions `newLength ..
int EvictBlocksBelow(long sequenceId, int keepFromPosition) Sliding-window KV retention: releases the leading KV blocks of sequenceId that hold only positions below keepFromPosition (the window lower bound), returning their memory to the pool so a long sequence’s KV footprint stays bounded by th…
int GetSequenceLength(long sequenceId) Gets the current length of a sequence.
int[] GetBlockTable(long sequenceId) Gets the block table for a sequence (for paged attention kernel).
void Dispose() Releases resources.
void FreeSequence(long sequenceId) Frees all cache blocks for a sequence.
void ReadKey(long sequenceId, int tokenPosition, int layer, Span<T> keyData) Reads key tensor for a token position.
void ReadKeyValueRange(long sequenceId, int startPosition, int count, int layer, Span<T> keyDst, Span<T> valueDst) Bulk-reads Key AND Value for a contiguous range of token positions [startPosition, startPosition+count) for one layer, under a SINGLE lock, into contiguous per-position slices of keyDst and valueDst (each laid out [count, NumHeads*Hea…
void ReadValue(long sequenceId, int tokenPosition, int layer, Span<T> valueData) Reads value tensor for a token position.
void WriteKey(long sequenceId, int tokenPosition, int layer, ReadOnlySpan<T> keyData) Writes key tensor for a token position.
void WriteValue(long sequenceId, int tokenPosition, int layer, ReadOnlySpan<T> valueData) Writes value tensor for a token position.