EmbeddingLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Represents an embedding layer that converts discrete token indices into dense vector representations.

Inheritance: LayerBase<T> → EmbeddingLayer<T>

Implements: IAuxiliaryLossLayer<T>, IShapeContract, ITokenEmbedding<T>

For Beginners

An embedding layer turns words or other symbols into lists of numbers that capture their meaning.

Imagine you have a dictionary where:

  • Each word has an ID number (like “cat” = 5, “dog” = 10)
  • The embedding layer gives each ID a unique “coordinate” in a multi-dimensional space
  • Words with similar meanings end up with similar coordinates

For example:

  • “Cat” might become [0.2, -0.5, 0.1, 0.8]
  • “Kitten” might become [0.25, -0.4, 0.15, 0.7]
  • “Computer” might become [-0.8, 0.2, 0.5, -0.3]

The embedding layer learns these representations during training, so that:

  • Similar words end up close to each other
  • Related concepts form clusters
  • The vectors capture meaningful semantic relationships

This allows neural networks to work with text and other discrete tokens in a way that captures their meaning and relationships.

How It Works

An embedding layer maps discrete tokens (represented as indices) to continuous vector representations. This is particularly useful for natural language processing tasks where words or tokens need to be represented as dense vectors that capture semantic relationships. Each token is assigned a unique vector in a high-dimensional space, allowing the model to learn meaningful representations.

Thread Safety: This layer is not thread-safe. Each layer instance maintains internal state during forward and backward passes. If you need concurrent execution, use separate layer instances per thread or synchronize access to shared instances.

Constructors

Constructor Summary
EmbeddingLayer(int vocabularySize, int embeddingDimension, double outputScale) Initializes a new instance of the EmbeddingLayer class.

Properties

Property Summary
IReadOnlyList<LayerPort> InputPorts { get; } Generated from [TensorPort] declarations.
IReadOnlyList<LayerPort> OutputPorts { get; } Generated from [TensorPort] declarations.
T AuxiliaryLossWeight { get; set; } Gets or sets the weight for embedding regularization.
bool IsInitialized { get; }
bool ScaleBySqrtDimension { get; set; } When true, the token-embedding lookup output is multiplied by sqrt(embeddingDimension) (Vaswani et al.
bool SupportsTraining { get; } Gets a value indicating whether this layer supports training.
bool UseAuxiliaryLoss { get; set; } Gets or sets whether to use auxiliary loss (embedding regularization) during training.

Methods

Method Summary
Dictionary<string, string> GetAuxiliaryLossDiagnostics() Gets diagnostic information about the embedding regularization.
Dictionary<string, string> GetDiagnostics() Gets diagnostic information about this component’s state and behavior.
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
LayerInputDomain GetInputDomain(int[] inputShape) Generated from the primary [TensorPort] declaration.
Matrix<T> GetTokenEmbeddings(IReadOnlyList<int> tokenIds)
T ComputeAuxiliaryLoss() Computes the auxiliary loss for the EmbeddingLayer, which is embedding regularization.
void ClearGradients()
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate) Updates the embedding matrix using the calculated gradients and the specified learning rate.
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).