TokenizerConfig
Options & Configuration · AiDotNet.Tokenization.HuggingFace
Configuration for HuggingFace tokenizers.
Constructors
| Constructor | Summary |
|---|---|
TokenizerConfig() |
Properties
| Property | Summary |
|---|---|
List<string> AdditionalSpecialTokens { get; set; } |
Gets or sets additional special tokens. |
bool DoLowerCase { get; set; } |
Gets or sets whether to lowercase input. |
int? ModelMaxLength { get; set; } |
Gets or sets the model max length. |
string BosToken { get; set; } |
|
string ClsToken { get; set; } |
|
string EosToken { get; set; } |
|
string MaskToken { get; set; } |
|
string MergesFile { get; set; } |
Gets or sets the merges file (for BPE). |
string ModelType { get; set; } |
Gets or sets the model type. |
string PadToken { get; set; } |
|
string SepToken { get; set; } |
|
string TokenizerClass { get; set; } |
Gets or sets the tokenizer type. |
string UnkToken { get; set; } |
Gets or sets special tokens. |
string VocabFile { get; set; } |
Gets or sets the vocabulary file. |