TransformerNormPlacement

Enums · AiDotNet.Enums

Where a transformer block applies layer normalization relative to each residual connection.

For Beginners

Each transformer sublayer (attention, feed-forward) adds its output back to its input. Pre-norm normalizes the sublayer’s input and leaves the running sum un-normalized; post-norm normalizes the sum after the addition, as the original Transformer did.

Fields

Field Summary
TransformerNormPlacement PostNorm y = LayerNorm(x + Sublayer(x)) (Vaswani et al.
TransformerNormPlacement PreNorm y = x + Sublayer(LayerNorm(x)) (Xiong et al.
int value__