TokenizerConfig

Options & Configuration · AiDotNet.Tokenization.HuggingFace

Configuration for HuggingFace tokenizers.

Constructors

Constructor Summary
TokenizerConfig()

Properties

Property Summary
List<string> AdditionalSpecialTokens { get; set; } Gets or sets additional special tokens.
bool DoLowerCase { get; set; } Gets or sets whether to lowercase input.
int? ModelMaxLength { get; set; } Gets or sets the model max length.
string BosToken { get; set; }
string ClsToken { get; set; }
string EosToken { get; set; }
string MaskToken { get; set; }
string MergesFile { get; set; } Gets or sets the merges file (for BPE).
string ModelType { get; set; } Gets or sets the model type.
string PadToken { get; set; }
string SepToken { get; set; }
string TokenizerClass { get; set; } Gets or sets the tokenizer type.
string UnkToken { get; set; } Gets or sets special tokens.
string VocabFile { get; set; } Gets or sets the vocabulary file.