CodeBertTokenizer
Models & Types · AiDotNet.Tokenization.CodeTokenization
CodeBERT-compatible tokenizer for program synthesis and code understanding tasks.
Combines WordPiece tokenization with code-aware preprocessing.
Constructors
| Constructor |
Summary |
CodeBertTokenizer(IVocabulary vocabulary, ProgrammingLanguage language, SpecialTokens specialTokens) |
Creates a new CodeBERT tokenizer. |
Properties
| Property |
Summary |
ITokenizer Tokenizer { get; } |
Gets the underlying tokenizer. |
Methods
| Method |
Summary |
TokenizationResult EncodeCodeAndNL(string code, string naturalLanguage, EncodingOptions options) |
Encodes code and natural language for CodeBERT. |
string Decode(List<int> tokenIds, bool skipSpecialTokens) |
Decodes token IDs back to code. |