TransformerEncoderLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
Represents a transformer encoder layer that processes sequences using self-attention and feed-forward networks.
Inheritance: LayerBase<T> → TransformerEncoderLayer<T>
Implements: IAuxiliaryLossLayer<T>, IShapeContract
For Beginners
This layer helps a neural network understand relationships between different elements in a sequence.
Think of it like a careful reader analyzing a paragraph:
- First, the reader looks at how each word relates to every other word (self-attention)
- Then, the reader processes this information to understand the meaning (feed-forward network)
For example, in the sentence “The animal didn’t cross the street because it was too wide”:
- The self-attention helps the network understand that “it” refers to “the street” (not “the animal”)
- The feed-forward network processes this contextual information for each word
This architecture is powerful for tasks like understanding text, analyzing time series, or processing any data where the relationships between elements matter.
How It Works
A transformer encoder layer is a fundamental building block of transformer-based models for sequence processing tasks. It consists of two main components: a self-attention mechanism that allows each position in a sequence to attend to all positions, and a feed-forward network that processes each position independently. Each component is followed by layer normalization and residual connections to facilitate training of deep networks.
Constructors
| Constructor | Summary |
|---|---|
TransformerEncoderLayer(int numHeads, int feedForwardDim) |
Lazy ctor: _embeddingSize is resolved from input.Shape[^1] on first Forward; the inner attention/FFN/norm sublayers are constructed then. |
TransformerEncoderLayer(int numHeads, int feedForwardDim, int embeddingSize) |
Eager-dimension ctor: _embeddingSize is known at construction time, so ParameterCount can declare a non-zero count immediately (no first-forward required). |
Properties
| Property | Summary |
|---|---|
T AuxiliaryLossWeight { get; set; } |
Gets or sets the weight for the auxiliary loss contribution. |
bool SupportsTraining { get; } |
The computation engine (CPU or GPU) for vectorized operations. |
bool UseAuxiliaryLoss { get; set; } |
Gets or sets a value indicating whether auxiliary loss is enabled for this layer. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetAuxiliaryLossDiagnostics() |
Gets diagnostic information about the auxiliary loss computation. |
Dictionary<string, string> GetDiagnostics() |
Gets diagnostic information about this component’s state and behavior. |
IReadOnlyList<ILayer<T>> GetSubLayers() |
|
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
T ComputeAuxiliaryLoss() |
Computes the auxiliary loss for this layer by aggregating losses from sublayers. |
void ClearGradients() |
|
void ResetState() |
Resets the internal state of the transformer encoder layer and all its sublayers. |
void UpdateParameters(T learningRate) |
Updates the parameters of all sublayers using the calculated gradients. |
void UpdateParametersGpu(IGpuOptimizerConfig config) |
Updates layer parameters using GPU-resident optimizer. |