ICodeTokenizationPipeline
Interfaces · AiDotNet.ProgramSynthesis.Tokenization
Defines a code tokenization pipeline that builds on the core tokenizer stack and adds code-oriented metadata.
Known implementations: CodeTokenizationPipeline
Methods
| Method | Summary |
|---|---|
CodeTokenizationResult Tokenize(string code, ProgramLanguage language, ITokenizer tokenizer, EncodingOptions options, string filePath) |
Tokenizes source code and records where each token sits in it. |