TokenizationResult

Models & Types · AiDotNet.Tokenization.Models

Represents the result of tokenizing text, including token IDs, tokens, and attention masks.

Constructors

Constructor Summary
TokenizationResult() Creates an empty tokenization result.
TokenizationResult(List<string> tokens, List<int> tokenIds) Creates a tokenization result with the specified tokens and IDs.

Properties

Property Summary
Dictionary<string, object> Metadata { get; set; } Gets or sets additional metadata.
List<ValueTuple<int, int>> Offsets { get; set; } Gets or sets character-level offsets for each token.
List<int> AttentionMask { get; set; } Gets or sets the attention mask (1 for real tokens, 0 for padding).
List<int> PositionIds { get; set; } Gets or sets the position IDs for positional embeddings.
List<int> TokenIds { get; set; } Gets or sets the token IDs.
List<int> TokenTypeIds { get; set; } Gets or sets the token type IDs (for models that support multiple segments).
List<string> Tokens { get; set; } Gets or sets the actual tokens (subword strings).
int Length { get; } Gets the number of tokens (excluding padding).
int TotalLength { get; } Gets the total number of token IDs (including padding).