BlockManagerConfig
Options & Configuration · AiDotNet.Inference.PagedAttention
Configuration for the block manager.
Constructors
| Constructor |
Summary |
BlockManagerConfig() |
|
Properties
| Property |
Summary |
bool UseGpuMemory { get; set; } |
Whether to use GPU memory. |
int BlockSize { get; set; } |
Number of tokens per block. |
int GpuDeviceId { get; set; } |
GPU device ID (if using GPU memory). |
int HeadDimension { get; set; } |
Dimension of each attention head. |
int NumBlocks { get; set; } |
Total number of blocks to allocate. |
int NumHeads { get; set; } |
Number of attention heads per layer. |
int NumLayers { get; set; } |
Number of transformer layers. |
long BytesPerBlock { get; } |
Bytes per block (calculated based on model configuration). |
long TotalMemoryBytes { get; } |
Total memory required in bytes. |
Methods
| Method |
Summary |
BlockManagerConfig ForModel(string modelName, long availableMemoryBytes, int blockSize) |
Creates a configuration for a specific model. |
BlockManagerConfig FromMemorySize(long availableBytes, int blockSize, int numLayers, int numHeads, int headDim) |
Creates a configuration from available memory size. |