PagedKVCache<T>
Models & Types · AiDotNet.Inference.PagedAttention
Paged key-value cache for efficient LLM serving memory management.
Implements: IDisposable
For Beginners
Traditional KV-cache is like reserving a whole hotel floor per guest.
PagedKVCache is like renting hotel rooms individually:
- Guest arrives: Get them a room (allocate 1 block)
- Guest needs more space: Give them another room (allocate more blocks)
- Guest leaves: Rooms become available (free blocks)
Benefits:
- 8-9x more sequences can fit in memory
- No wasted space for short sequences
- Efficient beam search with copy-on-write
How It Works
PagedKVCache implements the vLLM-style paged attention memory management system. Instead of pre-allocating contiguous memory for each sequence’s maximum length, it dynamically allocates fixed-size blocks as sequences grow.
Constructors
| Constructor | Summary |
|---|---|
PagedKVCache(PagedKVCacheConfig config) |
Creates a new paged KV cache. |
Properties
| Property | Summary |
|---|---|
BlockManager<T> BlockManager { get; } |
Gets the block manager. |
BlockTableManager<T> BlockTableManager { get; } |
Gets the block table manager. |
PagedKVCacheConfig Config { get; } |
Gets the configuration. |
int ActiveSequenceCount { get; } |
Gets the number of active sequences. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<T[]> GetStorageBlocks() |
Gets the underlying storage blocks (for GPU transfer). |
PagedKVCache<T> FromMemorySize(long availableBytes, int numLayers, int numHeads, int headDim, int blockSize) |
Creates a new paged KV cache from memory size. |
PagedKVCacheStats GetStats() |
Gets statistics about the cache. |
bool AllocateSequence(long sequenceId, int initialTokens) |
Allocates cache space for a new sequence. |
bool ExtendSequence(long sequenceId, int additionalTokens) |
Extends a sequence’s cache for additional tokens. |
bool ForkSequence(long sourceSequenceId, long newSequenceId) |
Forks a sequence’s cache for beam search. |
bool HasCapacityFor(long sequenceId, int additionalTokens) |
Checks if more tokens can be added to a sequence without new allocation. |
bool TruncateSequence(long sequenceId, int newLength) |
Rolls a sequence’s logical length back to newLength, discarding the KV at positions `newLength .. |
int EvictBlocksBelow(long sequenceId, int keepFromPosition) |
Sliding-window KV retention: releases the leading KV blocks of sequenceId that hold only positions below keepFromPosition (the window lower bound), returning their memory to the pool so a long sequence’s KV footprint stays bounded by th… |
int GetSequenceLength(long sequenceId) |
Gets the current length of a sequence. |
int[] GetBlockTable(long sequenceId) |
Gets the block table for a sequence (for paged attention kernel). |
void Dispose() |
Releases resources. |
void FreeSequence(long sequenceId) |
Frees all cache blocks for a sequence. |
void ReadKey(long sequenceId, int tokenPosition, int layer, Span<T> keyData) |
Reads key tensor for a token position. |
void ReadKeyValueRange(long sequenceId, int startPosition, int count, int layer, Span<T> keyDst, Span<T> valueDst) |
Bulk-reads Key AND Value for a contiguous range of token positions [startPosition, startPosition+count) for one layer, under a SINGLE lock, into contiguous per-position slices of keyDst and valueDst (each laid out [count, NumHeads*Hea… |
void ReadValue(long sequenceId, int tokenPosition, int layer, Span<T> valueData) |
Reads value tensor for a token position. |
void WriteKey(long sequenceId, int tokenPosition, int layer, ReadOnlySpan<T> keyData) |
Writes key tensor for a token position. |
void WriteValue(long sequenceId, int tokenPosition, int layer, ReadOnlySpan<T> valueData) |
Writes value tensor for a token position. |