CodeTokenizationResult
Models & Types · AiDotNet.ProgramSynthesis.Models
Represents the result of code-aware tokenization, including token IDs and token-to-source spans.
How It Works
This type is designed to reuse the existing tokenization stack (ITokenizer) while providing code-specific structural metadata (spans) that is useful for downstream tasks (search, review, diagnostics, etc.).
Constructors
| Constructor | Summary |
|---|---|
CodeTokenizationResult() |
Properties
| Property | Summary |
|---|---|
List<CodeAstEdge> AstEdges { get; set; } |
Optional relationships between AST nodes (when enabled). |
List<CodeAstNode> AstNodes { get; set; } |
Optional AST nodes extracted during tokenization (when enabled and supported for the language). |
List<CodeSpan> TokenSpans { get; set; } |
Best-effort mapping from each token to a span in the original source code. |
ProgramLanguage Language { get; set; } |
|
TokenizationResult Tokenization { get; set; } |
|
string AstUnavailableReason { get; set; } |
Why the result has no syntax tree although one was requested (for example, the provider’s native grammar could not load), or null. |
string FilePath { get; set; } |