SentencePieceTokenizer

Models & Types ยท AiDotNet.Tokenization.Algorithms

SentencePiece tokenizer implementation using Unigram language model. Used for multilingual models and language-agnostic tokenization.

Inheritance: TokenizerBase โ†’ SentencePieceTokenizer

Constructors

Constructor Summary
SentencePieceTokenizer(IVocabulary vocabulary, Dictionary<string, double> pieceScores, SpecialTokens specialTokens, bool treatWhitespaceAsSpecialToken) Creates a new SentencePiece tokenizer.

Methods

Method Summary
List<string> Tokenize(string text) Tokenizes text into SentencePiece tokens.
SentencePieceTokenizer Train(IEnumerable<string> corpus, int vocabSize, SpecialTokens specialTokens, double characterCoverage) Trains a SentencePiece tokenizer using Unigram language model.