BatchScheduler<T>

Models & Types · AiDotNet.Serving.ContinuousBatching

Schedules sequences for continuous batching based on priority and resource constraints.

For Beginners

The scheduler is like a restaurant host.

When new customers arrive (requests), the host must decide:

  • Who gets seated next? (priority)
  • How many can we serve at once? (batch size)
  • Do we have enough tables/kitchen capacity? (memory/compute)
  • Should we pause someone’s meal to serve urgent customers? (preemption)

The scheduler makes these decisions to maximize throughput while ensuring fairness and meeting priority requirements.

How It Works

The scheduler determines which sequences should be processed in each iteration, balancing priorities, fairness, and memory constraints.

Constructors

Constructor Summary
BatchScheduler() Creates a new batch scheduler with default configuration.
BatchScheduler(BatchSchedulerConfig config) Creates a new batch scheduler with the specified configuration.

Properties

Property Summary
BatchSchedulerConfig Config { get; } Gets the scheduler configuration.
int PreemptedCount { get; } Gets the number of preempted sequences waiting to resume.
int RunningCount { get; } Gets the number of sequences currently being processed.
int WaitingCount { get; } Gets the number of sequences currently waiting to be processed.

Methods

Method Summary
List<SequenceState<T>> GetRunningSequences() Gets all currently running sequences.
List<SequenceState<T>> ScheduleNextBatch() Schedules the next batch of sequences to process.
SchedulerStatistics GetStatistics() Gets statistics about the scheduler state.
bool CancelSequence(long sequenceId) Cancels a sequence, removing it from all queues.
void AddSequence(SequenceState<T> sequence) Adds a new sequence to the waiting queue.
void CompleteSequence(SequenceState<T> sequence) Marks a sequence as completed and removes it from the running set.
void PreemptSequence(SequenceState<T> sequence) Preempts a running sequence, moving it to the preempted queue.
void ReorderByPriority() Reorders running sequences by priority.