SpeculativeDecodingOptions

Options & Configuration · AiDotNet.Configuration

Tuning knobs for speculative decoding — the technique that speeds up autoregressive (GPT-style) text generation by letting a small, fast “draft” model guess the next few tokens and having the main model verify all of them in a single pass. These options control how speculation runs; which draft model does the guessing is chosen separately (see the builder’s ConfigureSpeculativeDecoding overloads, which accept an IDraftModel or wrap a whole model).

For Beginners

Generating text one token at a time is slow because the big model runs once per token. Speculative decoding uses a cheap “guesser” to propose several tokens at once, then the big model checks them all in one shot. Guesses that match are kept for free; wrong guesses are simply discarded, so the output is identical to normal generation — just faster (often 1.5–3×). These settings let you tune the trade-off between how far ahead to guess and how aggressive to be. The defaults are safe; leave them alone if you’re unsure.

How It Works

When left at its defaults, speculation uses a zero-cost N-gram / prompt-lookup draft (no extra model needed), so simply enabling it via SpeculativeDecoding.Enabled = true already helps repetitive text.

Constructors

Constructor Summary
SpeculativeDecodingOptions()

Properties

Property Summary
SpeculationPolicy SpeculationPolicy { get; set; } Gets or sets the policy for when speculative decoding should actually run at request time.
SpeculativeMethod SpeculativeMethod { get; set; } Gets or sets the speculative decoding method (the “style” of speculation).
bool Enabled { get; set; } Gets or sets whether speculative decoding is enabled.
bool UseTreeSpeculation { get; set; } Gets or sets whether to use tree-structured speculation (multiple candidate branches per step).
int SpeculationDepth { get; set; } Gets or sets the speculation depth (number of tokens the draft model guesses ahead each step).

Methods

Method Summary
SpeculativeDecodingOptions Clone() Returns a copy of these options.