AutoregressiveDecoder<T>
Helpers & Utilities · AiDotNet.NeuralNetworks.Generation
Generic autoregressive decode loop (#1632 / #95): the reusable “generate” driver the codebase
lacked — GPT4Vision / Blip / Flamingo each hand-rolled this loop. It owns the loop + EOS + token
feedback + RNG lifetime; the model supplies the per-step “embed the token, run the (KV-cached)
forward, return next-token logits” via the stepLogits delegate. Because the cached
attention layers append to the KV cache inside that forward, each step only pays for the new
token instead of recomputing the prefix (proven equivalent by KVCacheDecodeEquivalenceTests).
Methods
| Method | Summary |
|---|---|
IReadOnlyList<int> Decode(Func<int?, Vector<T>> stepLogits, int maxNewTokens, SamplingOptions options, Func<int, bool> isEndToken, Func<int, bool> suppressToken) |
Greedily/stochastically decodes up to maxNewTokens tokens. |