PositionalEncodingLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Represents a layer that adds positional encodings to input sequences.

Inheritance: LayerBase<T> → PositionalEncodingLayer<T>

Implements: IShapeContract

For Beginners

This layer adds information about position to your sequence data.

Think of it like numbering the words in a sentence:

  • Without position information, a model only knows which words are in the sentence
  • With position information, it knows which word comes first, second, third, etc.

For example, the sentences “dog bites man” and “man bites dog” contain the same words but have completely different meanings because of word order. Positional encoding helps models understand this difference.

The layer uses a clever mathematical pattern of sine and cosine waves to encode positions. This approach has several advantages:

  • It creates a unique pattern for each position
  • Similar positions have similar encodings (helpful for generalization)
  • It can potentially handle sequences longer than those seen during training
  • The encodings have consistent patterns that models can learn from

How It Works

The PositionalEncodingLayer adds position-dependent signals to input embeddings, which helps sequence models like Transformers understand the order of elements in a sequence. Since attention-based models have no inherent notion of sequence order, positional encodings provide this critical information. The encodings use sine and cosine functions of different frequencies to create unique position-dependent patterns.

Constructors

Constructor Summary
PositionalEncodingLayer(int maxSequenceLength, int embeddingSize) Initializes a new instance of the PositionalEncodingLayer class with the specified maximum sequence length and embedding size.

Properties

Property Summary
bool SupportsTraining { get; } The computation engine (CPU or GPU) for vectorized operations.

Methods

Method Summary
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
void ResetState() Resets the internal state of the positional encoding layer.