RoBERTaNER<T>
Models & Types · AiDotNet.NER.TransformerBased
RoBERTa-NER: Robustly Optimized BERT Approach with token classification for NER.
Inheritance: NeuralNetworkBase<T> → NERNeuralNetworkBase<T> → SequenceLabelingNERBase<T> → TransformerNERBase<T> → RoBERTaNER<T>
For Beginners
RoBERTa is a “better-trained BERT.” The architecture is identical, but RoBERTa was trained more carefully on more data, resulting in consistently better performance. Use RoBERTa-NER when you want the best accuracy from a standard transformer model.
How It Works
RoBERTa-NER (Liu et al., 2019 - “RoBERTa: A Robustly Optimized BERT Pretraining Approach”) fine-tunes a RoBERTa model for NER. RoBERTa improves upon BERT through better pre-training:
Key Improvements over BERT:
- No Next Sentence Prediction (NSP): Removes the NSP objective which was found unhelpful
- Dynamic masking: Different tokens are masked in each epoch (vs BERT’s static masking)
- Longer training: Trained on 160GB of text (10x more than BERT)
- Larger batch sizes: 8K sequences per batch for more stable training
- Byte-Pair Encoding (BPE): Uses BPE tokenization instead of WordPiece
Performance (CoNLL-2003):
- RoBERTa-base: ~92.8% F1 (vs BERT-base ~92.4%)
- RoBERTa-large: ~93.2% F1 (vs BERT-large ~92.8%)
Example
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 512, numClasses: 9);
var roBERTaNER = new RoBERTaNER<float>(architecture);
Constructors
| Constructor | Summary |
|---|---|
RoBERTaNER(NeuralNetworkArchitecture<T> architecture, TransformerNEROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a RoBERTa-NER model in native training mode. |
RoBERTaNER(NeuralNetworkArchitecture<T> architecture, string modelPath, TransformerNEROptions options) |
Creates a RoBERTa-NER model in native training mode. |