NeuralDraftModel<T>
Models & Types · AiDotNet.Inference.SpeculativeDecoding
Ready-to-use IDraftModel that wraps any next-token-logits function as the draft model for
speculative decoding. Supply a smaller/faster model’s forward pass (tokens -> logits) and this handles the
autoregressive drafting and sampling for you.
Implements: IDraftModel<T>
For Beginners
This class wraps a neural network (like a small transformer) to use as the “fast guesser” in speculative decoding. The neural network should be much smaller and faster than the main model you’re trying to accelerate. You give it a function that turns tokens into next-token scores; it does the rest.
Constructors
| Constructor | Summary |
|---|---|
NeuralDraftModel(Func<Vector<int>, Vector<T>> forwardFunc, int vocabSize, int maxDraftTokens, int? seed) |
Creates a neural draft model wrapper. |
Properties
| Property | Summary |
|---|---|
int MaxDraftTokens { get; } |
|
int VocabSize { get; } |
Methods
| Method | Summary |
|---|---|
DraftResult<T> GenerateDraft(Vector<int> inputTokens, int numDraftTokens, T temperature) |
|
void Reset() |