IDraftModel<T>
Interfaces · AiDotNet.Inference.SpeculativeDecoding
Interface for draft models used in speculative decoding. Implement this to plug a custom draft model into AiDotNet’s serving engine (e.g. a small distilled model, an EAGLE/Medusa head, or a domain-specific heuristic) so the target model verifies your drafts instead of the built-in N-gram prompt-lookup draft.
Known implementations: NGramDraftModel<T>, NeuralDraftModel<T>
For Beginners
Speculative decoding speeds up generation by having a small, fast “draft” model guess the next few tokens, which the big model then checks all at once. This interface is that draft model’s contract: given the tokens so far, return a handful of guessed next tokens (and their probabilities). AiDotNet ships an N-gram draft by default; implement this interface if you want to supply your own faster/smarter guesser.
How It Works
Draft models are small, fast models that generate candidate tokens for verification by the larger target model. They should be lightweight enough to generate multiple tokens with minimal latency.
Properties
| Property | Summary |
|---|---|
int MaxDraftTokens { get; } |
Gets the maximum number of tokens this draft model can generate in one call. |
int VocabSize { get; } |
Gets the vocabulary size of this model. |
Methods
| Method | Summary |
|---|---|
DraftResult<T> GenerateDraft(Vector<int> inputTokens, int numDraftTokens, T temperature) |
Generates draft tokens autoregressively. |
void Reset() |
Resets any internal state (e.g., KV cache). |