CharacterTokenizer

Models & Types ยท AiDotNet.Tokenization.Algorithms

Character-level tokenizer that splits text into individual characters. Useful for character-based language models and some RNN architectures.

Inheritance: TokenizerBase โ†’ CharacterTokenizer

Constructors

Constructor Summary
CharacterTokenizer(IVocabulary vocabulary, SpecialTokens specialTokens, bool lowercase, bool includeWhitespace) Creates a new character tokenizer.

Methods

Method Summary
CharacterTokenizer CreateAscii(SpecialTokens specialTokens, bool lowercase) Creates a character tokenizer with ASCII printable characters.
CharacterTokenizer Train(IEnumerable<string> corpus, SpecialTokens specialTokens, bool lowercase, int minFrequency) Trains a character tokenizer from a corpus.
List<string> Tokenize(string text) Tokenizes text into individual characters.