DistilBERTNER<T>

Models & Types · AiDotNet.NER.TransformerBased

DistilBERT-NER: Knowledge-distilled BERT for efficient Named Entity Recognition.

Inheritance: NeuralNetworkBase<T> → NERNeuralNetworkBase<T> → SequenceLabelingNERBase<T> → TransformerNERBase<T> → DistilBERTNER<T>

For Beginners

DistilBERT is a compressed version of BERT that runs 60% faster while keeping 97% of the accuracy. Think of it as a “student” model that learned from the “teacher” (BERT). Use DistilBERT-NER when you need fast NER with good (but not maximum) accuracy, especially for production deployments with latency requirements.

How It Works

DistilBERT-NER (Sanh et al., NeurIPS 2019 Workshop - “DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter”) uses knowledge distillation to create a compact BERT variant that retains 97% of BERT’s language understanding while being 60% faster.

Knowledge Distillation Process:

  • Teacher: Full BERT-base model (12 layers, 110M parameters)
  • Student: DistilBERT (6 layers, 66M parameters - 40% fewer)
  • Training signal: Combination of distillation loss (soft targets from teacher),

masked language modeling loss, and cosine embedding loss (align hidden states)

  • Architecture: Removes token-type embeddings and the pooler layer, keeps every other

transformer layer from BERT

Performance vs Efficiency:

  • 40% smaller than BERT-base (66M vs 110M parameters)
  • 60% faster inference (fewer transformer layers)
  • Retains 97% of BERT’s performance on GLUE benchmark
  • NER (CoNLL-2003): ~91.2% F1 (vs BERT-base ~92.4%)

Trade-offs: DistilBERT sacrifices ~1.2% F1 on NER compared to BERT-base, but gains significant speed and memory improvements. This makes it ideal for production deployments where latency and resource constraints matter more than the last percentage of accuracy.

Example

var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 512, numClasses: 9);
var distilBERTNER = new DistilBERTNER<float>(architecture);

Constructors

Constructor Summary
DistilBERTNER(NeuralNetworkArchitecture<T> architecture, TransformerNEROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a DistilBERT-NER model in native training mode.
DistilBERTNER(NeuralNetworkArchitecture<T> architecture, string modelPath, TransformerNEROptions options) Creates a DistilBERT-NER model in native training mode.