TabRNormalization

Enums · AiDotNet.Enums

The normalization layer TabR builds wherever the reference model calls Normalization(d_main): before every block but the first encoder block, before the key projection when mixer normalization is on, and at the start of the prediction head.

For Beginners

normalization rescales each hidden vector so training stays stable. Keep LayerNorm, which is what the paper uses, unless you are experimenting.

How It Works

The reference yandex-research/tabular-dl-tabr resolves the layer with getattr(torch.nn, normalization), so any torch.nn module taking the width is accepted. These are the ones with a meaning there.

Fields

Field Summary
TabRNormalization BatchNorm nn.BatchNorm1d(d_main), normalizing each feature over the batch, with running statistics at prediction.
TabRNormalization Identity nn.Identity: no normalization at all.
TabRNormalization LayerNorm nn.LayerNorm(d_main), normalizing each row over its features: the paper’s default.
int value__