BiasMode
Enums · AiDotNet.Enums
Whether a layer that can carry an additive bias should actually do so.
For Beginners
a bias lets a layer shift its output up or down. If the next step is
a normalization that already shifts, the bias has nothing left to do. Leave this at
Auto and the layer works it out; set it explicitly only to force the issue.
How It Works
A bias is redundant when the layer’s output is immediately normalized by something that applies its own learnable shift. The normalization subtracts the mean, which removes whatever constant the bias added, then adds its own beta in its place. Two parameters then describe one degree of freedom: the bias cannot receive a meaningful gradient, and it costs memory, checkpoint size and optimizer state for nothing.
Reference implementations usually hard-code this decision against the norm type they happen to
use. pix2pix (Isola et al., 2017) writes use_bias = norm_layer == nn.InstanceNorm2d, which
is right only because their InstanceNorm is built with affine=False and their BatchNorm
with affine=True. The same expression is wrong for InstanceNorm2d(affine=True),
which does supply a shift, and wrong for a scale-only normalization such as RMSNorm, which does
not. Auto asks the following layer instead of inferring from its type, so it stays
correct for normalizations those implementations never considered.
Fields
| Field | Summary |
|---|---|
BiasMode Always |
Always carry a bias, even under a normalization that already shifts. |
BiasMode Auto |
Decide from what follows: no bias when the next layer supplies a learnable shift, a bias otherwise. |
BiasMode Never |
Never carry a bias. |
BiasMode Unspecified |
Not stated. |
int value__ |