TransformerNormPlacement
Enums · AiDotNet.Enums
Where a transformer block applies layer normalization relative to each residual connection.
For Beginners
Each transformer sublayer (attention, feed-forward) adds its output back to its input. Pre-norm normalizes the sublayer’s input and leaves the running sum un-normalized; post-norm normalizes the sum after the addition, as the original Transformer did.
Fields
| Field | Summary |
|---|---|
TransformerNormPlacement PostNorm |
y = LayerNorm(x + Sublayer(x)) (Vaswani et al. |
TransformerNormPlacement PreNorm |
y = x + Sublayer(LayerNorm(x)) (Xiong et al. |
int value__ |