ContinuousBatcherConfig
Options & Configuration · AiDotNet.Serving.ContinuousBatching
Configuration for the continuous batcher.
Constructors
| Constructor |
Summary |
ContinuousBatcherConfig() |
|
Properties
| Property |
Summary |
BatchSchedulerConfig SchedulerConfig { get; set; } |
Scheduler configuration. |
SpeculationPolicy SpeculationPolicy { get; set; } |
Policy for when speculative decoding should run (default: Auto). |
SpeculativeMethod SpeculativeMethod { get; set; } |
Speculative decoding method to use (default: Auto). |
bool AutoStart { get; set; } |
Whether to automatically start the batcher when a request is submitted. |
bool EnableSpeculativeDecoding { get; set; } |
Whether to enable speculative decoding. |
bool SupportsBatchedPrefill { get; set; } |
Whether the paged incremental model accepts a multi-token forward that yields per-position logits (batched prefill / speculative verification). |
bool UseTreeSpeculation { get; set; } |
Whether to use tree-based speculation (multiple draft continuations). |
int EosTokenId { get; set; } |
End-of-sequence token ID. |
int IdleSleepMs { get; set; } |
Milliseconds to sleep when idle. |
int MaxContextLength { get; set; } |
Maximum number of tokens in context (prompt + generated). |
int MaxPrefillChunkTokens { get; set; } |
Maximum prompt tokens prefilled per step for CHUNKED prefill. |
int MaxTreeDepth { get; set; } |
Maximum depth of the speculation tree. |
int SpeculationDepth { get; set; } |
Number of tokens to draft ahead when speculative decoding is enabled. |
int TreeBranchFactor { get; set; } |
Branching factor for tree speculation (continuations explored per step). |
Methods
| Method |
Summary |
ContinuousBatcherConfig ForModel(string modelName, int maxBatchSize) |
Creates config for a specific model. |
Fields
| Field |
Summary |
int DefaultEosTokenId |
Default end-of-sequence token ID when none is configured. |