IGenerationTokenizer
Interfaces · AiDotNet.Agentic.Models.Local
The minimal tokenizer contract the local generation engine needs: turn text into token ids, turn token ids back into text, and know which token marks end-of-sequence.
Known implementations: TokenizerGenerationAdapter
For Beginners
Models don’t read text directly — they read numbers (token ids). This turns your prompt into those numbers, turns the model’s numbers back into readable text, and tells the engine the special “stop here” token so it knows when the model is done.
How It Works
This is intentionally narrower than the full ITokenizer so
the engine stays decoupled and trivially testable. TokenizerGenerationAdapter bridges a
real repo tokenizer to this seam.
Properties
| Property | Summary |
|---|---|
int EosTokenId { get; } |
Gets the id of the end-of-sequence token. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<int> Encode(string text) |
Encodes text into token ids. |
string Decode(IReadOnlyList<int> tokenIds) |
Decodes token ids back into text. |