WordCharEmbeddingLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Paper-faithful word + character embedding front-end for sequence-labeling NER models (Lample et al., NAACL 2016, “Neural Architectures for Named Entity Recognition”, §3).

Inheritance: LayerBase<T> → WordCharEmbeddingLayer<T>

Implements: IShapeContract

For Beginners

a network can’t read text directly — words must become numbers. This layer is the “reading” front-end: it looks up a learned vector for each word AND spells each word out letter-by-letter through a tiny LSTM, so the model can also guess at words it never saw. It glues the two together for every word and hands the result to the rest of the network.

How It Works

This composite layer turns a sentence of integer token/character indices into the dense per-token representation that the downstream word-level BiLSTM-CRF consumes. It implements the two complementary embedding streams from Lample et al. (2016):

  • Word embeddings: a learnable lookup table (initializable from pretrained GloVe

vectors) maps each word index to a dense vector — word identity and, when pretrained, semantic similarity.

  • Character embeddings: a small bidirectional LSTM reads each word’s character

sequence and produces a fixed-size vector per word — morphology (capitalization, prefixes/suffixes, the shape of out-of-vocabulary words) that the word lookup alone cannot represent.

The two streams are concatenated per token, exactly as in the paper, giving a representation of size wordEmbeddingDim + charHiddenDim.

Implementation note — embeddings as linear layers. An embedding lookup is mathematically a linear projection of a one-hot index vector, so each lookup table is implemented as a no-frills DenseLayer applied to a one-hot encoding. This is deliberate: it routes the table through the same gradient-tape-tracked matmul that trains every other dense layer, so the word and character tables actually learn end-to-end. (A raw integer gather is faster for large vocabularies but, in this engine’s eager training path, does not propagate gradients back to the table — which would leave the embeddings frozen at their random initialization.) For very large vocabularies the one-hot matmul costs more memory than a gather; for typical NER vocabularies it is fine.

Why this matters: feeding a BiLSTM-CRF pre-computed, identity-only embeddings (or hash-derived vectors) yields a model that can only memorize the training vocabulary, hallucinates on unseen words, and — with a process-randomized hash — produces non-deterministic output. Owning real learnable word and character tables is the standard fix.

Input contract. A single packed integer tensor of shape [sequenceLength, 1 + maxWordLength] per sentence: column 0 holds the word index, columns 1..maxWordLength hold that word’s character indices (zero-padded). Index 0 is reserved for padding. Output shape is [sequenceLength, wordEmbeddingDim + charHiddenDim].

Constructors

Constructor Summary
WordCharEmbeddingLayer(int wordVocabSize, int wordEmbeddingDim, int charVocabSize, int charEmbeddingDim, int charHiddenDim, int sequenceLength, int maxWordLength) Initializes a new WordCharEmbeddingLayer.

Properties

Property Summary
DenseLayer<T> WordEmbedding { get; } Gets the underlying word-embedding linear layer, exposed so a model can initialize it from pretrained vectors.
IReadOnlyList<LayerPort> InputPorts { get; } Generated from [TensorPort] declarations.
IReadOnlyList<LayerPort> OutputPorts { get; } Generated from [TensorPort] declarations.
bool SupportsTraining { get; }
int OutputEmbeddingDim { get; } Gets the dimensionality of the per-token output vector (wordEmbeddingDim + charHiddenDim) that feeds the downstream word-level BiLSTM.

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
LayerInputDomain GetInputDomain(int[] inputShape) Generated from the primary [TensorPort] declaration.
void ResetState()
void UpdateParameters(T learningRate)