SciBERTNER<T>
Models & Types · AiDotNet.NER.TransformerBased
SciBERT-NER: Scientific domain BERT for Named Entity Recognition in scientific literature.
Inheritance: NeuralNetworkBase<T> → NERNeuralNetworkBase<T> → SequenceLabelingNERBase<T> → TransformerNERBase<T> → SciBERTNER<T>
For Beginners
SciBERT is BERT pre-trained entirely on scientific papers. It has a specialized vocabulary designed for scientific terms. Use SciBERT-NER for extracting entities from scientific literature, especially when working across both biomedical and computer science domains. For purely biomedical NER, BioBERT or PubMedBERT may perform slightly better.
How It Works
SciBERT-NER (Beltagy et al., EMNLP 2019 - “SciBERT: A Pretrained Language Model for Scientific Text”) is BERT pre-trained from scratch on 1.14M scientific papers from Semantic Scholar, covering computer science (18%) and biomedical (82%) domains.
Key Differences from BioBERT:
- Pre-trained from scratch: SciBERT uses its own vocabulary (SciVocab) built from
scientific text, while BioBERT initializes from BERT’s vocabulary
- Domain-specific vocabulary: SciVocab (31K tokens) contains scientific terms that
BERT’s WordPiece vocabulary would split into many subwords
- Broader scientific coverage: Includes computer science papers, not just biomedical
Scientific NER Tasks:
- SciERC: Scientific entities (Task, Method, Metric, Material, Generic, OtherScientificTerm)
- JNLPBA: Biomedical entities (Protein, DNA, RNA, Cell Line, Cell Type) - 77.3% F1
- BC5CDR: Chemical and disease entities - 90.0% F1
- NCBI Disease: Disease mention recognition - 88.6% F1
Architecture: Same as BERT-base (12 layers, 768 hidden, 12 heads, 110M params) but with SciVocab tokenization. The key insight is that domain-specific vocabulary matters: “immunoglobulin” is one token in SciVocab but [“im”, “##mun”, “##og”, “##lo”, “##bul”, “##in”] in BERT.
Example
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 512, numClasses: 9);
var sciBERTNER = new SciBERTNER<float>(architecture);
Constructors
| Constructor | Summary |
|---|---|
SciBERTNER(NeuralNetworkArchitecture<T> architecture, TransformerNEROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a SciBERT-NER model in native training mode. |
SciBERTNER(NeuralNetworkArchitecture<T> architecture, string modelPath, TransformerNEROptions options) |
Creates a SciBERT-NER model in native training mode. |