NGramDraftModel<T>

Models & Types · AiDotNet.Inference.SpeculativeDecoding

A simple n-gram based draft model for speculative decoding. It is the zero-cost default “guesser”: it predicts the next token from the statistics of which tokens typically follow the recent ones, needing no second neural network and no GPU.

Implements: IDraftModel<T>

For Beginners

This is a very simple model that predicts the next word based on what typically follows the previous words.

For example, if “the” is often followed by “quick” in training data, then when we see “the”, this model might predict “quick”.

It’s not as good as a neural network, but it’s very fast! Speculative decoding uses it as the default “guesser” — you can pass one to the builder’s ConfigureSpeculativeDecoding if you want to control its order/seed, or supply your own IDraftModel (for example a NeuralDraftModel wrapping a smaller model) for smarter guesses.

Constructors

Constructor Summary
NGramDraftModel(int ngramSize, int vocabSize, int? seed) Creates an n-gram draft model.

Properties

Property Summary
int MaxDraftTokens { get; }
int VocabSize { get; }

Methods

Method Summary
DraftResult<T> GenerateDraft(Vector<int> inputTokens, int numDraftTokens, T temperature)
void Reset()
void Train(IEnumerable<Vector<int>> corpus) Trains the n-gram model on a corpus.