TokenizerBase
Base Classes · AiDotNet.Tokenization.Core
Base class for tokenizers providing common functionality.
Implements: ITokenizer
Known derived types: BpeTokenizer, CharacterTokenizer, CodeTokenizer, MidiTokenizer, PhonemeTokenizer, SentencePieceTokenizer, UnigramTokenizer, WordPieceTokenizer
Properties
| Property | Summary |
|---|---|
IVocabulary Vocabulary { get; set; } |
Gets the vocabulary. |
SpecialTokens SpecialTokens { get; set; } |
Gets the special tokens. |
int VocabularySize { get; } |
Gets the vocabulary size. |
Methods
| Method | Summary |
|---|---|
List<TokenizationResult> EncodeBatch(List<string> texts, EncodingOptions options) |
Encodes multiple texts into tokens. |
List<int> ConvertTokensToIds(List<string> tokens) |
Converts tokens to token IDs. |
List<string> ConvertIdsToTokens(List<int> ids) |
Converts token IDs to tokens. |
List<string> DecodeBatch(List<List<int>> tokenIdsBatch, bool skipSpecialTokens) |
Decodes multiple sequences of token IDs back into text. |
List<string> Tokenize(string text) |
Tokenizes text into subword tokens (must be implemented by derived classes). |
TokenizationResult Encode(string text, EncodingOptions options) |
Encodes text into tokens. |
string Decode(List<int> tokenIds, bool skipSpecialTokens) |
Decodes token IDs back into text. |