SpecialTokens
Models & Types · AiDotNet.Tokenization.Models
Represents special tokens used by tokenizers.
Constructors
| Constructor |
Summary |
SpecialTokens() |
|
Properties
| Property |
Summary |
List<string> AdditionalSpecialTokens { get; set; } |
Gets or sets additional special tokens. |
string BosToken { get; set; } |
Gets or sets the beginning of sequence token. |
string ClsToken { get; set; } |
Gets or sets the classification token (start of sequence). |
string EosToken { get; set; } |
Gets or sets the end of sequence token. |
string MaskToken { get; set; } |
Gets or sets the mask token. |
string PadToken { get; set; } |
Gets or sets the padding token. |
string SepToken { get; set; } |
Gets or sets the separation token. |
string UnkToken { get; set; } |
Gets or sets the unknown token. |
Methods
| Method |
Summary |
List<string> GetAllSpecialTokens() |
Gets all special tokens as a list. |
SpecialTokens Bert() |
Creates BERT-style special tokens. |
SpecialTokens Clip() |
Creates CLIP-style special tokens. |
SpecialTokens Default() |
Creates default special tokens (BERT-style). |
SpecialTokens Gpt() |
Creates GPT-style special tokens. |
SpecialTokens T5() |
Creates T5-style special tokens. |