RoBERTaNER<T>

Models & Types · AiDotNet.NER.TransformerBased

RoBERTa-NER: Robustly Optimized BERT Approach with token classification for NER.

Inheritance: NeuralNetworkBase<T> → NERNeuralNetworkBase<T> → SequenceLabelingNERBase<T> → TransformerNERBase<T> → RoBERTaNER<T>

For Beginners

RoBERTa is a “better-trained BERT.” The architecture is identical, but RoBERTa was trained more carefully on more data, resulting in consistently better performance. Use RoBERTa-NER when you want the best accuracy from a standard transformer model.

How It Works

RoBERTa-NER (Liu et al., 2019 - “RoBERTa: A Robustly Optimized BERT Pretraining Approach”) fine-tunes a RoBERTa model for NER. RoBERTa improves upon BERT through better pre-training:

Key Improvements over BERT:

  • No Next Sentence Prediction (NSP): Removes the NSP objective which was found unhelpful
  • Dynamic masking: Different tokens are masked in each epoch (vs BERT’s static masking)
  • Longer training: Trained on 160GB of text (10x more than BERT)
  • Larger batch sizes: 8K sequences per batch for more stable training
  • Byte-Pair Encoding (BPE): Uses BPE tokenization instead of WordPiece

Performance (CoNLL-2003):

  • RoBERTa-base: ~92.8% F1 (vs BERT-base ~92.4%)
  • RoBERTa-large: ~93.2% F1 (vs BERT-large ~92.8%)

Example

var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 512, numClasses: 9);
var roBERTaNER = new RoBERTaNER<float>(architecture);

Constructors

Constructor Summary
RoBERTaNER(NeuralNetworkArchitecture<T> architecture, TransformerNEROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a RoBERTa-NER model in native training mode.
RoBERTaNER(NeuralNetworkArchitecture<T> architecture, string modelPath, TransformerNEROptions options) Creates a RoBERTa-NER model in native training mode.