AutoregressiveDecoder<T>

Helpers & Utilities · AiDotNet.NeuralNetworks.Generation

Generic autoregressive decode loop (#1632 / #95): the reusable “generate” driver the codebase lacked — GPT4Vision / Blip / Flamingo each hand-rolled this loop. It owns the loop + EOS + token feedback + RNG lifetime; the model supplies the per-step “embed the token, run the (KV-cached) forward, return next-token logits” via the stepLogits delegate. Because the cached attention layers append to the KV cache inside that forward, each step only pays for the new token instead of recomputing the prefix (proven equivalent by KVCacheDecodeEquivalenceTests).

Methods

Method Summary
IReadOnlyList<int> Decode(Func<int?, Vector<T>> stepLogits, int maxNewTokens, SamplingOptions options, Func<int, bool> isEndToken, Func<int, bool> suppressToken) Greedily/stochastically decodes up to maxNewTokens tokens.