SpecialTokens

Models & Types · AiDotNet.Tokenization.Models

Represents special tokens used by tokenizers.

Constructors

Constructor Summary
SpecialTokens()

Properties

Property Summary
List<string> AdditionalSpecialTokens { get; set; } Gets or sets additional special tokens.
string BosToken { get; set; } Gets or sets the beginning of sequence token.
string ClsToken { get; set; } Gets or sets the classification token (start of sequence).
string EosToken { get; set; } Gets or sets the end of sequence token.
string MaskToken { get; set; } Gets or sets the mask token.
string PadToken { get; set; } Gets or sets the padding token.
string SepToken { get; set; } Gets or sets the separation token.
string UnkToken { get; set; } Gets or sets the unknown token.

Methods

Method Summary
List<string> GetAllSpecialTokens() Gets all special tokens as a list.
SpecialTokens Bert() Creates BERT-style special tokens.
SpecialTokens Clip() Creates CLIP-style special tokens.
SpecialTokens Default() Creates default special tokens (BERT-style).
SpecialTokens Gpt() Creates GPT-style special tokens.
SpecialTokens T5() Creates T5-style special tokens.