LearnedPositionalEmbeddingLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Adds a LEARNED per-position vector to a sequence of feature vectors — the position embedding used by BERT, ViT, BEiT and everything descended from them.

Inheritance: LayerBase<T> → LearnedPositionalEmbeddingLayer<T>

Implements: IShapeContract

For Beginners

A transformer sees a sequence as an unordered set unless you tell it where each item sits. This layer adds a small learned vector to each position, so the model can tell the first word from the fifth. “Learned” means those vectors are trained with everything else rather than computed from a formula.

How It Works

This is the trainable counterpart of PositionalEncodingLayer. That layer computes fixed sinusoids and declares SupportsTraining => false, which is right for the original Transformer (Vaswani et al. 2017) and wrong for the very large family that learns positions instead: BERT, ViT, BEiT, and the document models built on them all hold a [maxSequenceLength, dim] table and train it.

The distinction is not cosmetic. A sinusoid encodes position as a fixed function of the index, so the model can only use whatever relationships that function happens to expose. A learned table lets position 0 mean “start of document” and position 7 mean whatever the data says, which is why the models above chose it. Substituting sinusoids leaves the architecture unable to represent what the paper trained.

Several models in this repository declared exactly this table as a model FIELD — allocated, randomly initialised, counted as a parameter, serialized into every checkpoint and stepped by the optimizer — while a sinusoidal layer sat in the stack doing the work and nothing ever read the field. Both halves of that are fixed by having one trainable layer in the stack.

Constructors

Constructor Summary
LearnedPositionalEmbeddingLayer(int maxSequenceLength, int embeddingDim) Initializes a new LearnedPositionalEmbeddingLayer.

Properties

Property Summary
EmbeddingLayer<T> PositionTable { get; } Gets the position table, exposed so a model can load pretrained vectors.
bool SupportsTraining { get; }
int EmbeddingDim { get; } Width of each position vector, matching the features it is added to.
int MaxSequenceLength { get; } Longest sequence the table can index; later positions reuse the last row.

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
void ResetState()
void UpdateParameters(T learningRate)