BatchScheduler<T>
Models & Types · AiDotNet.Serving.ContinuousBatching
Schedules sequences for continuous batching based on priority and resource constraints.
For Beginners
The scheduler is like a restaurant host.
When new customers arrive (requests), the host must decide:
- Who gets seated next? (priority)
- How many can we serve at once? (batch size)
- Do we have enough tables/kitchen capacity? (memory/compute)
- Should we pause someone’s meal to serve urgent customers? (preemption)
The scheduler makes these decisions to maximize throughput while ensuring fairness and meeting priority requirements.
How It Works
The scheduler determines which sequences should be processed in each iteration, balancing priorities, fairness, and memory constraints.
Constructors
| Constructor | Summary |
|---|---|
BatchScheduler() |
Creates a new batch scheduler with default configuration. |
BatchScheduler(BatchSchedulerConfig config) |
Creates a new batch scheduler with the specified configuration. |
Properties
| Property | Summary |
|---|---|
BatchSchedulerConfig Config { get; } |
Gets the scheduler configuration. |
int PreemptedCount { get; } |
Gets the number of preempted sequences waiting to resume. |
int RunningCount { get; } |
Gets the number of sequences currently being processed. |
int WaitingCount { get; } |
Gets the number of sequences currently waiting to be processed. |
Methods
| Method | Summary |
|---|---|
List<SequenceState<T>> GetRunningSequences() |
Gets all currently running sequences. |
List<SequenceState<T>> ScheduleNextBatch() |
Schedules the next batch of sequences to process. |
SchedulerStatistics GetStatistics() |
Gets statistics about the scheduler state. |
bool CancelSequence(long sequenceId) |
Cancels a sequence, removing it from all queues. |
void AddSequence(SequenceState<T> sequence) |
Adds a new sequence to the waiting queue. |
void CompleteSequence(SequenceState<T> sequence) |
Marks a sequence as completed and removes it from the running set. |
void PreemptSequence(SequenceState<T> sequence) |
Preempts a running sequence, moving it to the preempted queue. |
void ReorderByPriority() |
Reorders running sequences by priority. |