CodeTokenizer
Models & Types ยท AiDotNet.Tokenization.CodeTokenization
Code-aware tokenizer that handles programming language constructs. Supports identifier splitting, keyword recognition, and language-specific patterns.
Inheritance: TokenizerBase โ CodeTokenizer
Constructors
| Constructor | Summary |
|---|---|
CodeTokenizer(ITokenizer baseTokenizer, ProgrammingLanguage language, bool splitIdentifiers) |
Creates a new code tokenizer. |
Methods
| Method | Summary |
|---|---|
List<string> Tokenize(string text) |
Tokenizes code with language-aware handling. |
TokenizationResult Encode(string text, EncodingOptions options) |
Encodes code into a tokenization result with best-effort character offsets. |