UnigramTokenizer
Models & Types ยท AiDotNet.Tokenization.Algorithms
Unigram Language Model tokenizer using probabilistic segmentation.
Inheritance: TokenizerBase โ UnigramTokenizer
Constructors
| Constructor | Summary |
|---|---|
UnigramTokenizer(IVocabulary vocabulary, Dictionary<string, double> tokenScores, SpecialTokens specialTokens, int maxTokenLength) |
Creates a new unigram tokenizer. |
Methods
| Method | Summary |
|---|---|
List<string> Tokenize(string text) |
Tokenizes text using Viterbi algorithm for optimal segmentation. |
UnigramTokenizer Train(IEnumerable<string> corpus, int vocabSize, SpecialTokens specialTokens) |
Trains a unigram tokenizer from a corpus. |