BatchSchedulerConfig

Options & Configuration · AiDotNet.Serving.ContinuousBatching

Configuration for the batch scheduler.

Constructors

Constructor Summary
BatchSchedulerConfig()

Properties

Property Summary
SchedulingPolicy Policy { get; set; } Scheduling policy to use.
bool AllowPreemption { get; set; } Whether to allow preempting lower-priority sequences.
int HeadDimension { get; set; } Dimension of each attention head (for memory estimation).
int MaxBatchSize { get; set; } Maximum number of sequences in a batch.
int MaxCacheSlots { get; set; } Maximum number of KV-cache slots available.
int NumHeads { get; set; } Number of attention heads (for memory estimation).
int NumLayers { get; set; } Number of transformer layers (for memory estimation).
long MaxMemoryBytes { get; set; } Maximum memory available for KV-cache (bytes).

Methods

Method Summary
BatchSchedulerConfig ForModel(string modelName, int maxBatchSize) Creates config for a specific model.