NeuralDraftModel<T>

Models & Types · AiDotNet.Inference.SpeculativeDecoding

Ready-to-use IDraftModel that wraps any next-token-logits function as the draft model for speculative decoding. Supply a smaller/faster model’s forward pass (tokens -> logits) and this handles the autoregressive drafting and sampling for you.

Implements: IDraftModel<T>

For Beginners

This class wraps a neural network (like a small transformer) to use as the “fast guesser” in speculative decoding. The neural network should be much smaller and faster than the main model you’re trying to accelerate. You give it a function that turns tokens into next-token scores; it does the rest.

Constructors

Constructor Summary
NeuralDraftModel(Func<Vector<int>, Vector<T>> forwardFunc, int vocabSize, int maxDraftTokens, int? seed) Creates a neural draft model wrapper.

Properties

Property Summary
int MaxDraftTokens { get; }
int VocabSize { get; }

Methods

Method Summary
DraftResult<T> GenerateDraft(Vector<int> inputTokens, int numDraftTokens, T temperature)
void Reset()