TabRNormalization
Enums · AiDotNet.Enums
The normalization layer TabR builds wherever the reference model calls Normalization(d_main): before
every block but the first encoder block, before the key projection when mixer normalization is on, and at the
start of the prediction head.
For Beginners
normalization rescales each hidden vector so training stays stable. Keep
LayerNorm, which is what the paper uses, unless you are experimenting.
How It Works
The reference yandex-research/tabular-dl-tabr resolves the layer with getattr(torch.nn, normalization),
so any torch.nn module taking the width is accepted. These are the ones with a meaning there.
Fields
| Field | Summary |
|---|---|
TabRNormalization BatchNorm |
nn.BatchNorm1d(d_main), normalizing each feature over the batch, with running statistics at prediction. |
TabRNormalization Identity |
nn.Identity: no normalization at all. |
TabRNormalization LayerNorm |
nn.LayerNorm(d_main), normalizing each row over its features: the paper’s default. |
int value__ |