CodeBertTokenizer

Models & Types · AiDotNet.Tokenization.CodeTokenization

CodeBERT-compatible tokenizer for program synthesis and code understanding tasks. Combines WordPiece tokenization with code-aware preprocessing.

Constructors

Constructor Summary
CodeBertTokenizer(IVocabulary vocabulary, ProgrammingLanguage language, SpecialTokens specialTokens) Creates a new CodeBERT tokenizer.

Properties

Property Summary
ITokenizer Tokenizer { get; } Gets the underlying tokenizer.

Methods

Method Summary
TokenizationResult EncodeCodeAndNL(string code, string naturalLanguage, EncodingOptions options) Encodes code and natural language for CodeBERT.
string Decode(List<int> tokenIds, bool skipSpecialTokens) Decodes token IDs back to code.