CodeTokenizationPipelineOptions
Options & Configuration · AiDotNet.ProgramSynthesis.Tokenization
Options for CodeTokenizationPipeline that control optional structural extraction.
Constructors
| Constructor | Summary |
|---|---|
CodeTokenizationPipelineOptions() |
Properties
| Property | Summary |
|---|---|
bool IncludeAst { get; set; } |
Gets or sets whether to extract a syntax tree (AST nodes and edges) alongside the tokens. |
int MaxAstNodes { get; set; } |
Gets or sets the most syntax-tree nodes extracted from one input; extraction stops once this many are collected. |