PubMedBERTNER<T>
Models & Types · AiDotNet.NER.TransformerBased
PubMedBERT-NER: PubMed domain-specific BERT pre-trained from scratch on biomedical text for NER.
Inheritance: NeuralNetworkBase<T> → NERNeuralNetworkBase<T> → SequenceLabelingNERBase<T> → TransformerNERBase<T> → PubMedBERTNER<T>
For Beginners
PubMedBERT is like BioBERT but even more specialized. While BioBERT started as a general English model and then read biomedical papers, PubMedBERT was built entirely from biomedical text from day one. It has a custom vocabulary designed for medical terms, making it the best choice for biomedical NER tasks like extracting drug names, disease names, gene names, and protein names from research papers and clinical text.
How It Works
PubMedBERT-NER (Gu et al., ACL 2021 - “Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing”) is BERT pre-trained entirely from scratch on PubMed abstracts, using a domain-specific vocabulary. Unlike BioBERT (which initializes from general BERT), PubMedBERT builds its vocabulary and model weights exclusively from biomedical text.
Key Difference from BioBERT:
- BioBERT: Starts from general BERT weights + vocabulary, then continues pre-training
on biomedical text. The vocabulary still contains many general-domain tokens.
- PubMedBERT: Pre-trained from scratch on PubMed only. Its vocabulary is 100%
biomedical, so terms like “immunoglobulin” or “phosphorylation” are single tokens instead of being split into many subwords.
Pre-training Data:
- PubMed abstracts only: ~3.1B words from 14M+ abstracts
- Custom WordPiece vocabulary built from PubMed text (28,895 tokens)
- No general-domain text (no Wikipedia, no BookCorpus)
Why From-Scratch Pre-training Matters: The vocabulary is the key. BioBERT inherits BERT’s general vocabulary where “phosphorylation” becomes [“ph”, “##os”, “##pho”, “##ry”, “##lation”] (5 tokens). In PubMedBERT, it’s a single token “phosphorylation”. This means:
- More efficient encoding of biomedical text
- Each token carries more semantic meaning
- Better representation of domain-specific terms
Performance (Biomedical NER):
- BC5CDR Chemical: ~94.1% F1 (vs BioBERT ~93.4%)
- BC5CDR Disease: ~87.8% F1 (vs BioBERT ~87.2%)
- NCBI Disease: ~89.8% F1 (vs BioBERT ~89.4%)
- BC2GM (Gene/Protein): ~88.1% F1 (vs BioBERT ~87.5%)
- JNLPBA: ~80.1% F1 (vs BioBERT ~79.3%)
PubMedBERT consistently outperforms BioBERT across all biomedical NER benchmarks, demonstrating that from-scratch domain-specific pre-training is superior to continued pre-training from a general model.
Example
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 512, numClasses: 9);
var pubMedBERTNER = new PubMedBERTNER<float>(architecture);
Constructors
| Constructor | Summary |
|---|---|
PubMedBERTNER(NeuralNetworkArchitecture<T> architecture, TransformerNEROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a PubMedBERT-NER model in native training mode. |
PubMedBERTNER(NeuralNetworkArchitecture<T> architecture, string modelPath, TransformerNEROptions options) |
Creates a PubMedBERT-NER model in native training mode. |