CodeTokenizationPipeline
Models & Types · AiDotNet.ProgramSynthesis.Tokenization
Default implementation of ICodeTokenizationPipeline.
Implements: ICodeTokenizationPipeline
Constructors
| Constructor | Summary |
|---|---|
CodeTokenizationPipeline() |
Creates a pipeline that uses the host’s fallback parser, if any, for syntax trees. |
CodeTokenizationPipeline(ICodeSyntaxProvider syntaxProvider) |
Creates a pipeline that extracts syntax trees with syntaxProvider. |
Methods
| Method | Summary |
|---|---|
CodeTokenizationResult Tokenize(string code, ProgramLanguage language, ITokenizer tokenizer, EncodingOptions options, string filePath) |
|
CodeTokenizationResult TokenizeWithStructure(string code, ProgramLanguage language, ITokenizer tokenizer, CodeTokenizationPipelineOptions pipelineOptions, EncodingOptions options, string filePath) |