SentencePieceTokenizer
Models & Types ยท AiDotNet.Tokenization.Algorithms
SentencePiece tokenizer implementation using Unigram language model. Used for multilingual models and language-agnostic tokenization.
Inheritance: TokenizerBase โ SentencePieceTokenizer
Constructors
| Constructor | Summary |
|---|---|
SentencePieceTokenizer(IVocabulary vocabulary, Dictionary<string, double> pieceScores, SpecialTokens specialTokens, bool treatWhitespaceAsSpecialToken) |
Creates a new SentencePiece tokenizer. |
Methods
| Method | Summary |
|---|---|
List<string> Tokenize(string text) |
Tokenizes text into SentencePiece tokens. |
SentencePieceTokenizer Train(IEnumerable<string> corpus, int vocabSize, SpecialTokens specialTokens, double characterCoverage) |
Trains a SentencePiece tokenizer using Unigram language model. |