ICodeTokenizationPipeline

Interfaces · AiDotNet.ProgramSynthesis.Tokenization

Defines a code tokenization pipeline that builds on the core tokenizer stack and adds code-oriented metadata.

Known implementations: CodeTokenizationPipeline

Methods

Method Summary
CodeTokenizationResult Tokenize(string code, ProgramLanguage language, ITokenizer tokenizer, EncodingOptions options, string filePath) Tokenizes source code and records where each token sits in it.