CodeTokenizer

Models & Types ยท AiDotNet.Tokenization.CodeTokenization

Code-aware tokenizer that handles programming language constructs. Supports identifier splitting, keyword recognition, and language-specific patterns.

Inheritance: TokenizerBase โ†’ CodeTokenizer

Constructors

Constructor Summary
CodeTokenizer(ITokenizer baseTokenizer, ProgrammingLanguage language, bool splitIdentifiers) Creates a new code tokenizer.

Methods

Method Summary
List<string> Tokenize(string text) Tokenizes code with language-aware handling.
TokenizationResult Encode(string text, EncodingOptions options) Encodes code into a tokenization result with best-effort character offsets.