BatchSchedulerConfig
Options & Configuration · AiDotNet.Serving.ContinuousBatching
Configuration for the batch scheduler.
Constructors
| Constructor |
Summary |
BatchSchedulerConfig() |
|
Properties
| Property |
Summary |
SchedulingPolicy Policy { get; set; } |
Scheduling policy to use. |
bool AllowPreemption { get; set; } |
Whether to allow preempting lower-priority sequences. |
int HeadDimension { get; set; } |
Dimension of each attention head (for memory estimation). |
int MaxBatchSize { get; set; } |
Maximum number of sequences in a batch. |
int MaxCacheSlots { get; set; } |
Maximum number of KV-cache slots available. |
int NumHeads { get; set; } |
Number of attention heads (for memory estimation). |
int NumLayers { get; set; } |
Number of transformer layers (for memory estimation). |
long MaxMemoryBytes { get; set; } |
Maximum memory available for KV-cache (bytes). |
Methods
| Method |
Summary |
BatchSchedulerConfig ForModel(string modelName, int maxBatchSize) |
Creates config for a specific model. |