TokenizerGenerationAdapter
Models & Types · AiDotNet.Agentic.Models.Local
Bridges a full repo ITokenizer to the engine’s minimal IGenerationTokenizer
seam, so any AiDotNet tokenizer can drive LocalEngineChatClient.
Implements: IGenerationTokenizer
For Beginners
The library’s tokenizers do a lot more than generation needs. This adapter exposes just the three things the generation loop uses (encode, decode, end-of-sequence id), so you can plug a real tokenizer into the local engine without it depending on the larger tokenizer surface.
Constructors
| Constructor | Summary |
|---|---|
TokenizerGenerationAdapter(ITokenizer tokenizer) |
Initializes a new adapter over the given tokenizer. |
Properties
| Property | Summary |
|---|---|
int EosTokenId { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<int> Encode(string text) |
|
string Decode(IReadOnlyList<int> tokenIds) |