TokenizationResult
Models & Types · AiDotNet.Tokenization.Models
Represents the result of tokenizing text, including token IDs, tokens, and attention masks.
Constructors
| Constructor | Summary |
|---|---|
TokenizationResult() |
Creates an empty tokenization result. |
TokenizationResult(List<string> tokens, List<int> tokenIds) |
Creates a tokenization result with the specified tokens and IDs. |
Properties
| Property | Summary |
|---|---|
Dictionary<string, object> Metadata { get; set; } |
Gets or sets additional metadata. |
List<ValueTuple<int, int>> Offsets { get; set; } |
Gets or sets character-level offsets for each token. |
List<int> AttentionMask { get; set; } |
Gets or sets the attention mask (1 for real tokens, 0 for padding). |
List<int> PositionIds { get; set; } |
Gets or sets the position IDs for positional embeddings. |
List<int> TokenIds { get; set; } |
Gets or sets the token IDs. |
List<int> TokenTypeIds { get; set; } |
Gets or sets the token type IDs (for models that support multiple segments). |
List<string> Tokens { get; set; } |
Gets or sets the actual tokens (subword strings). |
int Length { get; } |
Gets the number of tokens (excluding padding). |
int TotalLength { get; } |
Gets the total number of token IDs (including padding). |