TiedEmbeddingHeadLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
A language-model output head that shares its weights with the network’s token embedding: the logits are
h · Eᵀ, where E is the embedding table [vocabulary, hidden].
Inheritance: LayerBase<T> → TiedEmbeddingHeadLayer<T>
Implements: IShapeContract, ISharedParameterConsumerLayer<T>
For Beginners
The layer that turns words into vectors and the layer that turns vectors back into word scores use the same table, as in the original model, so training the model trains both at once.
How It Works
This is weight tying (Press & Wolf, 2017), used by Llama, Gemma and most checkpoints that set
tie_word_embeddings. The head owns no weights: it reads the embedding’s table on every forward, so a training
step moves one tensor, both uses contribute to its gradient, and the pair can never drift apart. Loading a tied
checkpoint into a separate dense head instead copies the table, which breaks the tie on the first update and holds
a second vocabulary × hidden matrix in memory.
The reference to the embedding layer is external state: it is not counted, optimized, cloned or saved here. The
network points it at the embedding’s current instance whenever its layer list is built or replaced
(BindToLayerGraph), and the embedding’s index in that list is what a saved model records.
Constructors
| Constructor | Summary |
|---|---|
TiedEmbeddingHeadLayer(EmbeddingLayer<T> embedding, double logitScale) |
Creates a head tied to embedding. |
TiedEmbeddingHeadLayer(int sourceLayerIndex, int vocabularySize, int embeddingDimension, double logitScale) |
Recreates a saved head, unbound. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
Methods
| Method | Summary |
|---|---|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
The last axis becomes the vocabulary; every leading axis passes through, as for a dense projection. |
IReadOnlyList<Tensor<T>> GetSharedTrainableParameters() |
|
Tensor<T> TryComputeFusedLoss(Tensor<T> input, Tensor<T> target, ILossFunction<T> loss) |
Computes the tied language-model head fused with CrossEntropyWithLogitsLoss (mean CE(input @ Eᵀ, ids)) without materializing [rows, vocabulary] logits on the autodiff tape. |
void ClearGradients() |
|
void ResetState() |