T5EncoderStack<T>
Layers · AiDotNet.NeuralNetworks.Layers
A T5 encoder stack whose blocks share one relative position bias table (Raffel et al. 2020, Sec. 2.1: “we share the position embedding parameters across all layers in our model”).
Inheritance: LayerBase<T> → T5EncoderStack<T>
Implements: IShapeContract
How It Works
The stack owns the single [numBuckets, numHeads] table and computes the [numHeads, S, S] bias once per forward pass, then hands it to every block’s attention as that pass’s input - the dataflow sharing of the reference implementation and of HuggingFace’s T5Stack, where only the first block holds the table.
No block holds the table or a reference to it. The earlier design passed the first block’s live tensor into every later block’s constructor, so the sharing was a pointer between layers: cloning could not rebuild it, and any path that copied layers independently would have given each block its own table. Here there is nothing to re-link. The table is one registered parameter, counted once, and the gradient of every block’s attention reaches it through the shared lookup.
Set shareRelativeBias to false for one table per block (the per-layer variant).
Constructors
| Constructor | Summary |
|---|---|
T5EncoderStack(int hiddenSize, int numLayers, int numHeads, int numBuckets, int maxDistance, bool shareRelativeBias, int? seed) |
Creates a stack of numLayers pre-LN T5 blocks. |
Properties
| Property | Summary |
|---|---|
IReadOnlyList<PreLNTransformerBlock<T>> Blocks { get; } |
The stack’s blocks, in forward order. |
bool SharesRelativeBias { get; } |
Whether the blocks share this stack’s single table. |
bool SupportsTraining { get; } |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
Tensor<T> GetRelativeBiasTable() |
The shared table, or an empty tensor when each block owns its own. |
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |