TokenizationConfig
Options & Configuration · AiDotNet.Tokenization.Configuration
Configuration options for tokenization in the prediction pipeline.
How It Works
For Beginners: Tokenization is the process of breaking text into smaller pieces (tokens) that a machine learning model can understand. Think of it like breaking a sentence into words, but sometimes words are further broken into subwords for better handling of unknown words.
Constructors
| Constructor | Summary |
|---|---|
TokenizationConfig() |
Properties
| Property | Summary |
|---|---|
EncodingOptions DefaultEncodingOptions { get; set; } |
Gets or sets the default encoding options for tokenization. |
bool AddSpecialTokens { get; set; } |
Gets or sets whether to automatically add special tokens (like [CLS], [SEP]) during encoding. |
bool EnableCaching { get; set; } |
Gets or sets whether to cache tokenization results for repeated inputs. |
bool EnableParallelBatchProcessing { get; set; } |
Gets or sets whether to use parallel processing for batch tokenization. |
bool Padding { get; set; } |
Gets or sets whether to pad sequences to the maximum length. |
bool ReturnAttentionMask { get; set; } |
Gets or sets whether to return attention masks. |
bool ReturnTokenTypeIds { get; set; } |
Gets or sets whether to return token type IDs (for models like BERT with multiple segments). |
bool Truncation { get; set; } |
Gets or sets whether to truncate sequences that exceed max length. |
int ParallelBatchThreshold { get; set; } |
Gets or sets the minimum batch size to trigger parallel processing. |
int? MaxLength { get; set; } |
Gets or sets the maximum sequence length for tokenization. |
string CacheDirectory { get; set; } |
Gets or sets an optional local cache directory used when downloading a pretrained tokenizer specified via PretrainedModelOrPath. |
string PaddingSide { get; set; } |
Gets or sets the side on which to pad sequences (“left” or “right”). |
string PretrainedModelOrPath { get; set; } |
Gets or sets an optional HuggingFace Hub model repository ID or local directory path used to load a pretrained tokenizer when no explicit ITokenizer instance is supplied to AiModelBuilder.ConfigureTokenizer(...). |
string TruncationSide { get; set; } |
Gets or sets the side on which to truncate sequences (“left” or “right”). |
Methods
| Method | Summary |
|---|---|
TokenizationConfig ForBert(int maxLength) |
Creates a configuration suitable for BERT-style models. |
TokenizationConfig ForCode(int maxLength) |
Creates a configuration suitable for code tokenization. |
TokenizationConfig ForGpt(int maxLength) |
Creates a configuration suitable for GPT-style models. |