CharacterTokenizer
Models & Types ยท AiDotNet.Tokenization.Algorithms
Character-level tokenizer that splits text into individual characters. Useful for character-based language models and some RNN architectures.
Inheritance: TokenizerBase โ CharacterTokenizer
Constructors
| Constructor | Summary |
|---|---|
CharacterTokenizer(IVocabulary vocabulary, SpecialTokens specialTokens, bool lowercase, bool includeWhitespace) |
Creates a new character tokenizer. |
Methods
| Method | Summary |
|---|---|
CharacterTokenizer CreateAscii(SpecialTokens specialTokens, bool lowercase) |
Creates a character tokenizer with ASCII printable characters. |
CharacterTokenizer Train(IEnumerable<string> corpus, SpecialTokens specialTokens, bool lowercase, int minFrequency) |
Trains a character tokenizer from a corpus. |
List<string> Tokenize(string text) |
Tokenizes text into individual characters. |