CodeTokenizationPipeline

Models & Types · AiDotNet.ProgramSynthesis.Tokenization

Default implementation of ICodeTokenizationPipeline.

Implements: ICodeTokenizationPipeline

Constructors

Constructor Summary
CodeTokenizationPipeline() Creates a pipeline that uses the host’s fallback parser, if any, for syntax trees.
CodeTokenizationPipeline(ICodeSyntaxProvider syntaxProvider) Creates a pipeline that extracts syntax trees with syntaxProvider.

Methods

Method Summary
CodeTokenizationResult Tokenize(string code, ProgramLanguage language, ITokenizer tokenizer, EncodingOptions options, string filePath)
CodeTokenizationResult TokenizeWithStructure(string code, ProgramLanguage language, ITokenizer tokenizer, CodeTokenizationPipelineOptions pipelineOptions, EncodingOptions options, string filePath)