CodeTokenizationResult

Models & Types · AiDotNet.ProgramSynthesis.Models

Represents the result of code-aware tokenization, including token IDs and token-to-source spans.

How It Works

This type is designed to reuse the existing tokenization stack (ITokenizer) while providing code-specific structural metadata (spans) that is useful for downstream tasks (search, review, diagnostics, etc.).

Constructors

Constructor Summary
CodeTokenizationResult()

Properties

Property Summary
List<CodeAstEdge> AstEdges { get; set; } Optional relationships between AST nodes (when enabled).
List<CodeAstNode> AstNodes { get; set; } Optional AST nodes extracted during tokenization (when enabled and supported for the language).
List<CodeSpan> TokenSpans { get; set; } Best-effort mapping from each token to a span in the original source code.
ProgramLanguage Language { get; set; }
TokenizationResult Tokenization { get; set; }
string AstUnavailableReason { get; set; } Why the result has no syntax tree although one was requested (for example, the provider’s native grammar could not load), or null.
string FilePath { get; set; }