TiedEmbeddingLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

A token embedding whose table also projects hidden states back to token logits (weight tying, Press and Wolf 2017), as T5’s shared input/output embedding does.

Inheritance: LayerBase<T> → TiedEmbeddingLayer<T>

Implements: IShapeContract

For Beginners

The table that turns a word id into a vector is reused, transposed, to score every word for the next position, so the model learns one set of word vectors instead of two.

How It Works

Forward looks rows up for token ids (any shape of integer-valued ids; the output adds a trailing dimension axis); Logits multiplies hidden states by the transposed table. Both use the same parameter, so its gradient collects from both ends.

Rows start at N(0, 1), as nn.Embedding does; T5 initializes its shared table at N(0, 1) too (initializer factor 1).

Constructors

Constructor Summary
TiedEmbeddingLayer(int vocabularySize, int dimension) Creates the layer.

Properties

Property Summary
IReadOnlyList<LayerPort> InputPorts { get; } Generated from [TensorPort] declarations.
IReadOnlyList<LayerPort> OutputPorts { get; } Generated from [TensorPort] declarations.
bool SupportsTraining { get; }
int Dimension { get; } Embedding width.
int VocabularySize { get; } Number of tokens.

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
LayerInputDomain GetInputDomain(int[] inputShape) Generated from the primary [TensorPort] declaration.
Tensor<T> Logits(Tensor<T> hidden) Token logits [..., vocabulary] of hidden states [..., dimension]: h · tableᵀ.
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).