TinyBERTNER<T>
Models & Types · AiDotNet.NER.TransformerBased
TinyBERT-NER: Two-stage distilled BERT for ultra-efficient Named Entity Recognition.
Inheritance: NeuralNetworkBase<T> → NERNeuralNetworkBase<T> → SequenceLabelingNERBase<T> → TransformerNERBase<T> → TinyBERTNER<T>
For Beginners
TinyBERT is one of the smallest BERT models available. It uses a sophisticated two-step learning process to compress BERT down to a fraction of its size while retaining useful accuracy. Use TinyBERT-NER for edge deployment, mobile applications, or when you need the fastest possible NER with reasonable accuracy.
How It Works
TinyBERT-NER (Jiao et al., EMNLP 2020 Findings - “TinyBERT: Distilling BERT for Natural Language Understanding”) uses a novel two-stage knowledge distillation framework to create an extremely compact BERT model (~7.5x smaller and ~9.4x faster than BERT-base).
Two-Stage Distillation:
- Stage 1 - General Distillation: Distill BERT’s general language knowledge using
a large unlabeled corpus. Transfers attention matrices, hidden states, and embedding layer knowledge from teacher to student.
- Stage 2 - Task-Specific Distillation: Fine-tune the student on task-specific data
(e.g., NER) using both the labeled data and the teacher’s task-specific knowledge.
Distillation Losses (Layer-by-Layer):
- Embedding loss: MSE between teacher and student embedding layers
- Attention loss: MSE between teacher and student attention matrices
- Hidden state loss: MSE between teacher and student hidden representations
- Prediction loss: Cross-entropy between teacher and student output distributions
TinyBERT-4L Architecture:
- 4 transformer layers (vs 12 in BERT-base)
- 312 hidden dimension (vs 768 in BERT-base)
- 12 attention heads (maintained for knowledge transfer)
- ~14.5M parameters (vs 110M in BERT-base)
Performance:
- NER (CoNLL-2003): ~89.5% F1 (vs BERT-base ~92.4%)
- 7.5x smaller than BERT-base
- 9.4x faster inference
Example
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 512, numClasses: 9);
var tinyBERTNER = new TinyBERTNER<float>(architecture);
Constructors
| Constructor | Summary |
|---|---|
TinyBERTNER(NeuralNetworkArchitecture<T> architecture, TransformerNEROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a TinyBERT-NER model in native training mode. |
TinyBERTNER(NeuralNetworkArchitecture<T> architecture, string modelPath, TransformerNEROptions options) |
Creates a TinyBERT-NER model in native training mode. |