ITokenizer

Interfaces · AiDotNet.Tokenization.Interfaces

Interface for text tokenizers.

Known implementations: TokenizerBase

Properties

Property Summary
IVocabulary Vocabulary { get; } Gets the vocabulary.
SpecialTokens SpecialTokens { get; } Gets the special tokens.
int VocabularySize { get; } Gets the vocabulary size.

Methods

Method Summary
List<TokenizationResult> EncodeBatch(List<string> texts, EncodingOptions options) Encodes multiple texts into tokens.
List<int> ConvertTokensToIds(List<string> tokens) Converts tokens to token IDs.
List<string> ConvertIdsToTokens(List<int> ids) Converts token IDs to tokens.
List<string> DecodeBatch(List<List<int>> tokenIdsBatch, bool skipSpecialTokens) Decodes multiple sequences of token IDs back into text.
List<string> Tokenize(string text) Tokenizes text into subword tokens (without converting to IDs).
TokenizationResult Encode(string text, EncodingOptions options) Encodes text into tokens.
string Decode(List<int> tokenIds, bool skipSpecialTokens) Decodes token IDs back into text.