IIncrementalCausalLanguageModel<T>
Interfaces · AiDotNet.Agentic.Models.Local
An ICausalLanguageModel that supports incremental (KV-cached) decoding: it processes the
prompt once, caches the per-position state, and then advances one token at a time without recomputing the
whole sequence. This is the fast path for autoregressive generation.
Implements: ICausalLanguageModel<T>
Known implementations: MambaCausalLanguageModel<T>, NeuralNetworkCausalLanguageModel<T>, OnnxFullModelAdapter<T>, WatermarkedIncrementalCausalLanguageModel<T>
For Beginners
Without caching, predicting each new word re-reads the entire conversation — slower and slower as it grows. With caching, the model remembers the work it already did and only looks at the one new word each time. This interface is how a model advertises “I can do the fast, remember-as-you-go version”; the engine uses it automatically when available and falls back to the simple way otherwise.
How It Works
The base NextTokenLogits re-feeds the full context each step, which
is correct but O(n²) over a generation. A model that maintains a key/value cache implements this interface
so LocalEngineChatClient can drive it incrementally: StartSequence primes
the cache with the prompt and returns the first next-token logits, then each AppendToken
feeds a single new token and returns the following logits. ResetCache clears state between
independent generations.
Methods
| Method | Summary |
|---|---|
Vector<T> AppendToken(int tokenId) |
Appends a single token to the cached context and returns the logits for the token after it. |
Vector<T> StartSequence(IReadOnlyList<int> promptTokenIds) |
Processes the prompt, populating the cache, and returns the logits for the first generated token. |
void ResetCache() |
Clears any cached decoding state, so the next StartSequence begins fresh. |