BlockManagerConfig

Options & Configuration · AiDotNet.Inference.PagedAttention

Configuration for the block manager.

Constructors

Constructor Summary
BlockManagerConfig()

Properties

Property Summary
bool UseGpuMemory { get; set; } Whether to use GPU memory.
int BlockSize { get; set; } Number of tokens per block.
int GpuDeviceId { get; set; } GPU device ID (if using GPU memory).
int HeadDimension { get; set; } Dimension of each attention head.
int NumBlocks { get; set; } Total number of blocks to allocate.
int NumHeads { get; set; } Number of attention heads per layer.
int NumLayers { get; set; } Number of transformer layers.
long BytesPerBlock { get; } Bytes per block (calculated based on model configuration).
long TotalMemoryBytes { get; } Total memory required in bytes.

Methods

Method Summary
BlockManagerConfig ForModel(string modelName, long availableMemoryBytes, int blockSize) Creates a configuration for a specific model.
BlockManagerConfig FromMemorySize(long availableBytes, int blockSize, int numLayers, int numHeads, int headDim) Creates a configuration from available memory size.