SpERTNER<T>
Models & Types · AiDotNet.NER.SpanBased
SpERT: Span-based Entity and Relation Transformer for joint entity and relation extraction.
Inheritance: NeuralNetworkBase<T> → NERNeuralNetworkBase<T> → SequenceLabelingNERBase<T> → SpanBasedNERBase<T> → SpERTNER<T>
For Beginners
SpERT looks at all possible groups of consecutive words (spans) in a sentence and classifies each as an entity type or non-entity. Unlike BiLSTM-CRF which labels each word one at a time, SpERT considers entire phrases at once. It can also find relationships between entities (e.g., “born in” between a person and a location).
How It Works
SpERT (Eberts and Ulges, ECAI 2020 - “Span-based Joint Entity and Relation Extraction with Transformer Pre-training”) performs joint entity recognition and relation extraction using a span-based approach built on top of a pre-trained transformer encoder.
Architecture Overview:
- Transformer Encoder: Pre-trained BERT encodes the input sentence into contextual
token representations
- Span Representation: For each candidate span (i, j), the representation is:
span(i,j) = [h_i; h_j; maxpool(h_i:h_j); width_embedding] where h_i, h_j are boundary tokens, maxpool is over the span content, and width_embedding encodes the span length
- Entity Classifier: A feedforward network classifies each span representation
into entity types or non-entity
- Relation Classifier: For each pair of predicted entity spans, a relation
classifier predicts the relation type using concatenated span representations plus context between the entities
Negative Sampling: Since most spans are non-entities and most entity pairs have no relation, SpERT uses careful negative sampling during training. The ratio of negative to positive samples is a key hyperparameter (typically 100:1 for entities).
Performance:
- CoNLL-2004: 86.3% entity F1, 72.9% relation F1
- ADE dataset: 89.3% entity F1, 79.2% relation F1
- SciERC: 70.3% entity F1, 48.4% relation F1
Key Insight: By operating on spans rather than individual tokens, SpERT avoids BIO label constraint issues and naturally handles multi-token entities. The joint entity-relation extraction ensures that entity and relation decisions are mutually informed.
Example
var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 512, numClasses: 9);
var spERTNER = new SpERTNER<float>(architecture);
Constructors
| Constructor | Summary |
|---|---|
SpERTNER(NeuralNetworkArchitecture<T> architecture, SpanBasedNEROptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) |
Creates a SpERT model in native training mode. |
SpERTNER(NeuralNetworkArchitecture<T> architecture, string modelPath, SpanBasedNEROptions options) |
Creates a SpERT model in native training mode. |