TokenizationConfig

Options & Configuration · AiDotNet.Tokenization.Configuration

Configuration options for tokenization in the prediction pipeline.

How It Works

For Beginners: Tokenization is the process of breaking text into smaller pieces (tokens) that a machine learning model can understand. Think of it like breaking a sentence into words, but sometimes words are further broken into subwords for better handling of unknown words.

Constructors

Constructor Summary
TokenizationConfig()

Properties

Property Summary
EncodingOptions DefaultEncodingOptions { get; set; } Gets or sets the default encoding options for tokenization.
bool AddSpecialTokens { get; set; } Gets or sets whether to automatically add special tokens (like [CLS], [SEP]) during encoding.
bool EnableCaching { get; set; } Gets or sets whether to cache tokenization results for repeated inputs.
bool EnableParallelBatchProcessing { get; set; } Gets or sets whether to use parallel processing for batch tokenization.
bool Padding { get; set; } Gets or sets whether to pad sequences to the maximum length.
bool ReturnAttentionMask { get; set; } Gets or sets whether to return attention masks.
bool ReturnTokenTypeIds { get; set; } Gets or sets whether to return token type IDs (for models like BERT with multiple segments).
bool Truncation { get; set; } Gets or sets whether to truncate sequences that exceed max length.
int ParallelBatchThreshold { get; set; } Gets or sets the minimum batch size to trigger parallel processing.
int? MaxLength { get; set; } Gets or sets the maximum sequence length for tokenization.
string CacheDirectory { get; set; } Gets or sets an optional local cache directory used when downloading a pretrained tokenizer specified via PretrainedModelOrPath.
string PaddingSide { get; set; } Gets or sets the side on which to pad sequences (“left” or “right”).
string PretrainedModelOrPath { get; set; } Gets or sets an optional HuggingFace Hub model repository ID or local directory path used to load a pretrained tokenizer when no explicit ITokenizer instance is supplied to AiModelBuilder.ConfigureTokenizer(...).
string TruncationSide { get; set; } Gets or sets the side on which to truncate sequences (“left” or “right”).

Methods

Method Summary
TokenizationConfig ForBert(int maxLength) Creates a configuration suitable for BERT-style models.
TokenizationConfig ForCode(int maxLength) Creates a configuration suitable for code tokenization.
TokenizationConfig ForGpt(int maxLength) Creates a configuration suitable for GPT-style models.