SequenceLabelingNERBase<T>

Base Classes · AiDotNet.NER.SequenceLabeling

Base class for sequence labeling NER models that assign a BIO label to each token in a sequence.

Inheritance: NeuralNetworkBase<T> → NERNeuralNetworkBase<T> → SequenceLabelingNERBase<T>

Known derived types: BiLSTMCRF<T>, CNNBiLSTMCRF<T>, LSTMCRF<T>, SpanBasedNERBase<T>, TransformerNERBase<T>, WordCharBiLSTMCRF<T>

For Beginners

Sequence labeling NER processes text one word at a time and assigns a label to each word. The labels use a special coding scheme called BIO:

  • B = “Begin” - marks the first word of an entity
  • I = “Inside” - marks continuation words of an entity
  • O = “Outside” - marks words that aren’t part of any entity

For example, in “John Smith works at Google”:

  • “John” -> B-PER (Beginning of a Person name)
  • “Smith” -> I-PER (Inside the same Person name)
  • “works” -> O (not an entity)
  • “at” -> O (not an entity)
  • “Google” -> B-ORG (Beginning of an Organization name)

The key advantage of this approach is that it naturally handles multi-word entities like “New York City” (B-LOC, I-LOC, I-LOC).

How It Works

Sequence labeling is the most common approach to Named Entity Recognition, where each token in a sentence is assigned a label from a predefined set using the BIO (Begin, Inside, Outside) tagging scheme. This base class provides the task-specific functionality shared by all sequence labeling NER models, analogous to how VideoSuperResolutionBase<T> provides shared functionality for all video super-resolution models.

The BIO scheme works as follows:

  • B-TYPE: Beginning of an entity of the given type (e.g., B-PER for the first token of a person name)
  • I-TYPE: Inside (continuation of) an entity (e.g., I-PER for subsequent tokens of a person name)
  • O: Outside any entity (regular words like verbs, prepositions, articles)

For example: “Barack Obama was born in Honolulu”

  • Barack -> B-PER (beginning of a person entity)
  • Obama -> I-PER (inside the same person entity)
  • was -> O (not an entity)
  • born -> O (not an entity)
  • in -> O (not an entity)
  • Honolulu -> B-LOC (beginning of a location entity)

This base class provides:

  • Label sequence prediction (abstract, implemented by concrete models)
  • Emission score computation (the per-token, per-label scores before CRF decoding)
  • Argmax decoding fallback for models without CRF
  • Label index to label name conversion utilities
  • CRF toggle for enabling/disabling structured prediction

Derived classes implement specific architectures like BiLSTM-CRF, BERT-NER, SpanNER, etc.

Properties

Property Summary
bool UseCRF { get; set; } Gets or sets whether to use CRF (Conditional Random Field) decoding for label sequence prediction.
string[] LabelNames { get; set; } Gets the label names corresponding to each label index in the BIO tagging scheme.

Methods

Method Summary
IReadOnlyList<string[]> DecodeLabelsBatch(Tensor<T> batchLabelIndices) Converts batched predicted label indices to human-readable BIO label name strings.
Tensor<T> PredictEmissions(Tensor<T> tokenEmbeddings) Returns the raw per-token emission scores (shape [sequenceLength, numLabels], or batched [batch, sequenceLength, numLabels]) produced by the encoder BEFORE CRF/Viterbi decoding.
Tensor<T> PredictLabels(Tensor<T> tokenEmbeddings) Predicts the optimal BIO label sequence for input token embeddings.
string[] DecodeLabels(Tensor<T> labelIndices) Converts predicted label indices to human-readable BIO label name strings.
void Train(Tensor<T> input, Tensor<T> expectedOutput)