BlockManager<T>

Models & Types · AiDotNet.Inference.PagedAttention

Manages physical memory blocks for PagedAttention KV cache.

For Beginners

Think of memory management like a parking lot.

Traditional KV-cache: Each car (request) gets a reserved section of spaces equal to the maximum parking time, even if they leave early. Wasteful!

PagedAttention: Cars share a common pool of parking spaces. When they arrive, they get spaces from the free pool. When they leave, spaces return to the pool.

Benefits:

  • No wasted space for requests shorter than max length
  • Can serve more requests with the same memory
  • Memory is only allocated when needed

How It Works

The BlockManager maintains a pool of fixed-size memory blocks that can be dynamically allocated and freed. This enables efficient memory utilization by avoiding pre-allocation of maximum sequence length for each request.

Constructors

Constructor Summary
BlockManager(BlockManagerConfig config) Creates a new block manager with the specified configuration.
BlockManager(int totalMemoryBytes, int blockSize, int numLayers, int numHeads, int headDim) Creates a block manager for a specific model configuration.

Properties

Property Summary
BlockManagerConfig Config { get; } Gets the configuration.
double MemoryUtilization { get; } Gets the memory utilization (0-1).
int AllocatedBlockCount { get; } Gets the number of allocated blocks.
int FreeBlockCount { get; } Gets the number of free blocks available.
int TotalBlocks { get; } Gets the total number of blocks.

Methods

Method Summary
BlockManagerStats GetStats() Gets statistics about the block manager.
bool CanAllocate(int count) Checks if the manager can allocate the specified number of blocks.
int AllocateBlock() Allocates a single block.
int BlocksForTokens(int numTokens) Calculates how many blocks are needed for the specified number of tokens.
int CopyOnWrite(int blockId, Action<int, int> copyData) Performs copy-on-write for a block if it has multiple references.
int GetReferenceCount(int blockId) Gets the reference count for a block.
int TokensForBlocks(int numBlocks) Calculates how many tokens can fit in the specified number of blocks.
int[] AllocateBlocks(int count) Allocates multiple blocks.
void AddReference(int blockId) Increments the reference count for a block (for copy-on-write).
void FreeBlock(int blockId) Frees a single block.
void FreeBlocks(IEnumerable<int> blockIds) Frees multiple blocks.
void Reset() Resets the block manager, freeing all blocks.