BertEmbeddingLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
Implements BERT’s parallel word, learned-position, and token-type embedding front-end.
Inheritance: LayerBase<T> → BertEmbeddingLayer<T>
Implements: IShapeContract
How It Works
RoBERTa (Liu et al. 2019; Hugging Face RobertaEmbeddings) uses the same block with one difference: the
position of a token is counted over the NON-padding tokens and offset past the padding index,
position = cumsum(id != pad) * (id != pad) + pad, so real tokens start at pad + 1 and every
padding token reads the padding row. Pass paddingTokenId to get that numbering; leave it negative for
BERT’s 0 .. L-1.
Constructors
| Constructor | Summary |
|---|---|
BertEmbeddingLayer(int vocabularySize, int hiddenSize, int maxSequenceLength, int tokenTypeVocabularySize, double dropoutProbability, double layerNormEpsilon, int paddingTokenId) |
Creates a canonical BERT embedding block. |
Properties
| Property | Summary |
|---|---|
EmbeddingLayer<T> PositionEmbedding { get; } |
Learned absolute-position lookup table. |
EmbeddingLayer<T> TokenTypeEmbedding { get; } |
Token-type (segment) lookup table. |
EmbeddingLayer<T> WordEmbedding { get; } |
Word lookup table. |
IReadOnlyList<LayerPort> InputPorts { get; } |
Generated from [TensorPort] declarations. |
IReadOnlyList<LayerPort> OutputPorts { get; } |
Generated from [TensorPort] declarations. |
LayerNormalizationLayer<T> Normalization { get; } |
The LayerNorm applied to the summed embeddings. |
bool SupportsTraining { get; } |
|
int PaddingTokenId { get; } |
The padding token id that RoBERTa-style positions count around, or a negative value for BERT positions `0 .. |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
|
LayerInputDomain GetInputDomain(int[] inputShape) |
Generated from the primary [TensorPort] declaration. |
Tensor<T> Forward(Inputs<T> inputs) |
|
void ResetState() |