ELECTRANER<T>
Models & Types · AiDotNet.NER.TransformerBased
ELECTRA-NER: Efficiently Learning an Encoder that Classifies Token Replacements Accurately for NER.
Inheritance: NeuralNetworkBase<T> → NERNeuralNetworkBase<T> → SequenceLabelingNERBase<T> → TransformerNERBase<T> → ELECTRANER<T>
For Beginners
ELECTRA is a smart, efficient transformer that learns faster than BERT. While BERT learns by filling in blank words (like a fill-in-the-blank test), ELECTRA learns by detecting fake words (like a fact-checker). This is more efficient because ELECTRA learns from every word in every sentence, not just the few blanked-out words.
Use ELECTRA-NER when:
- You want good accuracy with limited compute budget
- You need a smaller model that still performs well
- Training efficiency is important
How It Works
ELECTRA-NER (Clark et al., ICLR 2020 - “ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators”) uses a novel replaced token detection (RTD) pre-training objective instead of masked language modeling. ELECTRA is particularly efficient because:
Key Innovation - Replaced Token Detection: Instead of masking 15% of tokens and predicting them (like BERT), ELECTRA:
- A small “generator” network (like a tiny BERT) predicts masked tokens
- Some predicted tokens are plausible but wrong replacements
- The main “discriminator” network learns to detect which tokens were replaced
- Every token position provides a training signal (vs only 15% for BERT)
This means ELECTRA learns from ALL tokens in every sentence, making it 4x more efficient than BERT at the same compute budget. An ELECTRA-small model matches BERT-base performance using only 1/4 of the compute.
Performance (CoNLL-2003):
- ELECTRA-small: ~91.5% F1 (matching BERT-base with 1/4 compute)
- ELECTRA-base: ~92.6% F1
- ELECTRA-large: ~93.3% F1
Example
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 512, numClasses: 9);
var eLECTRANER = new ELECTRANER<float>(architecture);
Constructors
| Constructor | Summary |
|---|---|
ELECTRANER(NeuralNetworkArchitecture<T> architecture, TransformerNEROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates an ELECTRA-NER model in native training mode. |
ELECTRANER(NeuralNetworkArchitecture<T> architecture, string modelPath, TransformerNEROptions options) |
Creates an ELECTRA-NER model in native training mode. |