BlockManager<T>
Models & Types · AiDotNet.Inference.PagedAttention
Manages physical memory blocks for PagedAttention KV cache.
For Beginners
Think of memory management like a parking lot.
Traditional KV-cache: Each car (request) gets a reserved section of spaces equal to the maximum parking time, even if they leave early. Wasteful!
PagedAttention: Cars share a common pool of parking spaces. When they arrive, they get spaces from the free pool. When they leave, spaces return to the pool.
Benefits:
- No wasted space for requests shorter than max length
- Can serve more requests with the same memory
- Memory is only allocated when needed
How It Works
The BlockManager maintains a pool of fixed-size memory blocks that can be dynamically allocated and freed. This enables efficient memory utilization by avoiding pre-allocation of maximum sequence length for each request.
Constructors
| Constructor | Summary |
|---|---|
BlockManager(BlockManagerConfig config) |
Creates a new block manager with the specified configuration. |
BlockManager(int totalMemoryBytes, int blockSize, int numLayers, int numHeads, int headDim) |
Creates a block manager for a specific model configuration. |
Properties
| Property | Summary |
|---|---|
BlockManagerConfig Config { get; } |
Gets the configuration. |
double MemoryUtilization { get; } |
Gets the memory utilization (0-1). |
int AllocatedBlockCount { get; } |
Gets the number of allocated blocks. |
int FreeBlockCount { get; } |
Gets the number of free blocks available. |
int TotalBlocks { get; } |
Gets the total number of blocks. |
Methods
| Method | Summary |
|---|---|
BlockManagerStats GetStats() |
Gets statistics about the block manager. |
bool CanAllocate(int count) |
Checks if the manager can allocate the specified number of blocks. |
int AllocateBlock() |
Allocates a single block. |
int BlocksForTokens(int numTokens) |
Calculates how many blocks are needed for the specified number of tokens. |
int CopyOnWrite(int blockId, Action<int, int> copyData) |
Performs copy-on-write for a block if it has multiple references. |
int GetReferenceCount(int blockId) |
Gets the reference count for a block. |
int TokensForBlocks(int numBlocks) |
Calculates how many tokens can fit in the specified number of blocks. |
int[] AllocateBlocks(int count) |
Allocates multiple blocks. |
void AddReference(int blockId) |
Increments the reference count for a block (for copy-on-write). |
void FreeBlock(int blockId) |
Frees a single block. |
void FreeBlocks(IEnumerable<int> blockIds) |
Frees multiple blocks. |
void Reset() |
Resets the block manager, freeing all blocks. |