BiLSTMCRF<T>

Models & Types · AiDotNet.NER.SequenceLabeling

BiLSTM-CRF: Bidirectional LSTM with Conditional Random Field for Named Entity Recognition.

Inheritance: NeuralNetworkBase<T> → NERNeuralNetworkBase<T> → SequenceLabelingNERBase<T> → BiLSTMCRF<T>

Implements: INERModel<T>

For Beginners

BiLSTM-CRF is the go-to model for finding names, places, organizations, and other entities in text. Think of it as a two-step process:

Step 1 - Reading with Context (BiLSTM): The model reads each word while considering the words around it. It reads the sentence both forward and backward, like reading a mystery novel twice - once normally and once from the end. This gives each word a rich understanding of its context. For example, “Apple” in “Apple Inc. announced…” gets a different representation than “apple” in “She ate an apple.”

Step 2 - Making Consistent Labels (CRF): After understanding each word’s context, the CRF ensures the labels make sense as a sequence. It’s like a spell-checker for entity labels. Without CRF, the model might label “Obama” as I-ORG after labeling “Barack” as B-PER, which doesn’t make sense. The CRF knows that I-ORG can’t follow B-PER and would correct this to “Barack”=B-PER, “Obama”=I-PER.

**Example:**When to use BiLSTM-CRF:

  • Named entity recognition (people, organizations, locations, etc.)
  • Part-of-speech tagging
  • Chunking and shallow parsing
  • Any sequence labeling task where label dependencies matter

Key advantage over simpler approaches: The CRF layer typically improves F1 score by 1-2% over independent per-token classification, and more importantly, it guarantees that the output label sequence is always structurally valid.

How It Works

BiLSTM-CRF (Huang et al., 2015; Lample et al., NAACL 2016) is the foundational neural architecture for Named Entity Recognition that combines two powerful, complementary components into a unified model:

1. Bidirectional LSTM (BiLSTM): A pair of Long Short-Term Memory networks that process the token sequence in both directions simultaneously. The forward LSTM reads left-to-right (e.g., “Barack” -> “Obama” -> “was” -> “born” -> “in” -> “Honolulu”), capturing preceding context for each token. The backward LSTM reads right-to-left, capturing following context. Their hidden states are merged at each position, giving the model a complete view of the entire sentence context around every token. This bidirectional context is crucial for NER because entity recognition often depends on both left and right context (e.g., “Washington” could be a person, location, or organization depending on surrounding words).

2. Conditional Random Field (CRF): A structured prediction layer that models label-level transition dependencies to produce globally optimal label sequences. While the BiLSTM produces per-token scores for each possible label (called “emission scores”), the CRF adds a learned transition matrix that captures which label transitions are valid. For example, it learns that I-PER (Inside Person) can only follow B-PER (Begin Person) or I-PER, never B-ORG (Begin Organization). During inference, the Viterbi algorithm efficiently finds the highest-scoring label sequence that respects all transition constraints.

Architecture Pipeline:

  • Token embeddings (e.g., 100d GloVe vectors) are fed into the BiLSTM encoder
  • BiLSTM produces contextualized hidden states for each token position
  • A linear projection maps hidden states to emission scores (one score per label per token)
  • Dropout regularization prevents overfitting during training
  • CRF layer combines emission scores with learned transition scores to decode the optimal label sequence

Mathematical Formulation: For input sequence x = (x_1, …, x_n) and label sequence y = (y_1, …, y_n):

  • Emission score: e(x_t, y_t) = W * h_t + b, where h_t is the BiLSTM hidden state at position t
  • Transition score: T[y_{t-1}, y_t] from the learned CRF transition matrix
  • Sequence score: S(x, y) = SUM_t [e(x_t, y_t) + T[y_{t-1}, y_t]]
  • Training objective: maximize log P(y|x) = S(x, y) - log(SUM_{y’} exp(S(x, y’)))
  • Inference: y* = argmax_y S(x, y), solved efficiently by Viterbi algorithm in O(n * L^2) time

This model serves as the golden example for the NER model family in AiDotNet, establishing the architectural patterns (base class hierarchy, layer initialization via LayerHelper, dual ONNX/native mode, serialization, etc.) that all other NER models follow.

References:

  • “Bidirectional LSTM-CRF Models for Sequence Tagging” (Huang, Xu, Yu, 2015)
  • “Neural Architectures for Named Entity Recognition” (Lample et al., NAACL 2016)
  • “End-to-end Sequence Labeling via Bi-directional LSTM-CNNs-CRF” (Ma and Hovy, ACL 2016)

Example

var architecture = new NeuralNetworkArchitecture<float>(inputFeatures: 512, numClasses: 9);
var biLSTMCRF = new BiLSTMCRF<float>(architecture);

Constructors

Constructor Summary
BiLSTMCRF(NeuralNetworkArchitecture<T> architecture, BiLSTMCRFOptions options, IGradientBasedOptimizer<T, Tensor<T>, Tensor<T>> optimizer) Creates a BiLSTM-CRF model in native training mode with full forward/backward pass support.
BiLSTMCRF(NeuralNetworkArchitecture<T> architecture, string modelPath, BiLSTMCRFOptions options) Creates a BiLSTM-CRF model in native training mode with full forward/backward pass support.

Properties

Property Summary
int[] ExpectedInputShape { get; } Gets the expected input tensor shape as [maxSequenceLength, embeddingDimension].

Methods

Method Summary
Dictionary<string, Tensor<T>> GetNamedLayerActivations(Tensor<T> input)
ModelMetadata<T> GetModelMetadata() Returns metadata describing this model for serialization, logging, and model registry purposes.
Tensor<T> PredictLabels(Tensor<T> tokenEmbeddings) Predicts the optimal BIO label sequence for input token embeddings using the full BiLSTM-CRF pipeline (or ONNX inference if in ONNX mode).