UnigramTokenizer

Models & Types ยท AiDotNet.Tokenization.Algorithms

Unigram Language Model tokenizer using probabilistic segmentation.

Inheritance: TokenizerBase โ†’ UnigramTokenizer

Constructors

Constructor Summary
UnigramTokenizer(IVocabulary vocabulary, Dictionary<string, double> tokenScores, SpecialTokens specialTokens, int maxTokenLength) Creates a new unigram tokenizer.

Methods

Method Summary
List<string> Tokenize(string text) Tokenizes text using Viterbi algorithm for optimal segmentation.
UnigramTokenizer Train(IEnumerable<string> corpus, int vocabSize, SpecialTokens specialTokens) Trains a unigram tokenizer from a corpus.