SpeculativeDecodingOptions
Options & Configuration · AiDotNet.Configuration
Tuning knobs for speculative decoding — the technique that speeds up autoregressive (GPT-style) text
generation by letting a small, fast “draft” model guess the next few tokens and having the main model
verify all of them in a single pass. These options control how speculation runs; which draft
model does the guessing is chosen separately (see the builder’s ConfigureSpeculativeDecoding overloads,
which accept an IDraftModel or wrap a whole model).
For Beginners
Generating text one token at a time is slow because the big model runs once per token. Speculative decoding uses a cheap “guesser” to propose several tokens at once, then the big model checks them all in one shot. Guesses that match are kept for free; wrong guesses are simply discarded, so the output is identical to normal generation — just faster (often 1.5–3×). These settings let you tune the trade-off between how far ahead to guess and how aggressive to be. The defaults are safe; leave them alone if you’re unsure.
How It Works
When left at its defaults, speculation uses a zero-cost N-gram / prompt-lookup draft (no extra model needed),
so simply enabling it via SpeculativeDecoding.Enabled = true already helps repetitive text.
Constructors
| Constructor | Summary |
|---|---|
SpeculativeDecodingOptions() |
Properties
| Property | Summary |
|---|---|
SpeculationPolicy SpeculationPolicy { get; set; } |
Gets or sets the policy for when speculative decoding should actually run at request time. |
SpeculativeMethod SpeculativeMethod { get; set; } |
Gets or sets the speculative decoding method (the “style” of speculation). |
bool Enabled { get; set; } |
Gets or sets whether speculative decoding is enabled. |
bool UseTreeSpeculation { get; set; } |
Gets or sets whether to use tree-structured speculation (multiple candidate branches per step). |
int SpeculationDepth { get; set; } |
Gets or sets the speculation depth (number of tokens the draft model guesses ahead each step). |
Methods
| Method | Summary |
|---|---|
SpeculativeDecodingOptions Clone() |
Returns a copy of these options. |