TokenizerGenerationAdapter

Models & Types · AiDotNet.Agentic.Models.Local

Bridges a full repo ITokenizer to the engine’s minimal IGenerationTokenizer seam, so any AiDotNet tokenizer can drive LocalEngineChatClient.

Implements: IGenerationTokenizer

For Beginners

The library’s tokenizers do a lot more than generation needs. This adapter exposes just the three things the generation loop uses (encode, decode, end-of-sequence id), so you can plug a real tokenizer into the local engine without it depending on the larger tokenizer surface.

Constructors

Constructor Summary
TokenizerGenerationAdapter(ITokenizer tokenizer) Initializes a new adapter over the given tokenizer.

Properties

Property Summary
int EosTokenId { get; }

Methods

Method Summary
IReadOnlyList<int> Encode(string text)
string Decode(IReadOnlyList<int> tokenIds)