ContinuousBatcherConfig

Options & Configuration · AiDotNet.Serving.ContinuousBatching

Configuration for the continuous batcher.

Constructors

Constructor Summary
ContinuousBatcherConfig()

Properties

Property Summary
BatchSchedulerConfig SchedulerConfig { get; set; } Scheduler configuration.
SpeculationPolicy SpeculationPolicy { get; set; } Policy for when speculative decoding should run (default: Auto).
SpeculativeMethod SpeculativeMethod { get; set; } Speculative decoding method to use (default: Auto).
bool AutoStart { get; set; } Whether to automatically start the batcher when a request is submitted.
bool EnableSpeculativeDecoding { get; set; } Whether to enable speculative decoding.
bool SupportsBatchedPrefill { get; set; } Whether the paged incremental model accepts a multi-token forward that yields per-position logits (batched prefill / speculative verification).
bool UseTreeSpeculation { get; set; } Whether to use tree-based speculation (multiple draft continuations).
int EosTokenId { get; set; } End-of-sequence token ID.
int IdleSleepMs { get; set; } Milliseconds to sleep when idle.
int MaxContextLength { get; set; } Maximum number of tokens in context (prompt + generated).
int MaxPrefillChunkTokens { get; set; } Maximum prompt tokens prefilled per step for CHUNKED prefill.
int MaxTreeDepth { get; set; } Maximum depth of the speculation tree.
int SpeculationDepth { get; set; } Number of tokens to draft ahead when speculative decoding is enabled.
int TreeBranchFactor { get; set; } Branching factor for tree speculation (continuations explored per step).

Methods

Method Summary
ContinuousBatcherConfig ForModel(string modelName, int maxBatchSize) Creates config for a specific model.

Fields

Field Summary
int DefaultEosTokenId Default end-of-sequence token ID when none is configured.