BertEmbeddingLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Implements BERT’s parallel word, learned-position, and token-type embedding front-end.

Inheritance: LayerBase<T> → BertEmbeddingLayer<T>

Implements: IShapeContract

How It Works

RoBERTa (Liu et al. 2019; Hugging Face RobertaEmbeddings) uses the same block with one difference: the position of a token is counted over the NON-padding tokens and offset past the padding index, position = cumsum(id != pad) * (id != pad) + pad, so real tokens start at pad + 1 and every padding token reads the padding row. Pass paddingTokenId to get that numbering; leave it negative for BERT’s 0 .. L-1.

Constructors

Constructor Summary
BertEmbeddingLayer(int vocabularySize, int hiddenSize, int maxSequenceLength, int tokenTypeVocabularySize, double dropoutProbability, double layerNormEpsilon, int paddingTokenId) Creates a canonical BERT embedding block.

Properties

Property Summary
EmbeddingLayer<T> PositionEmbedding { get; } Learned absolute-position lookup table.
EmbeddingLayer<T> TokenTypeEmbedding { get; } Token-type (segment) lookup table.
EmbeddingLayer<T> WordEmbedding { get; } Word lookup table.
IReadOnlyList<LayerPort> InputPorts { get; } Generated from [TensorPort] declarations.
IReadOnlyList<LayerPort> OutputPorts { get; } Generated from [TensorPort] declarations.
LayerNormalizationLayer<T> Normalization { get; } The LayerNorm applied to the summed embeddings.
bool SupportsTraining { get; }
int PaddingTokenId { get; } The padding token id that RoBERTa-style positions count around, or a negative value for BERT positions `0 ..

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
LayerInputDomain GetInputDomain(int[] inputShape) Generated from the primary [TensorPort] declaration.
Tensor<T> Forward(Inputs<T> inputs)
void ResetState()