BiasMode

Enums · AiDotNet.Enums

Whether a layer that can carry an additive bias should actually do so.

For Beginners

a bias lets a layer shift its output up or down. If the next step is a normalization that already shifts, the bias has nothing left to do. Leave this at Auto and the layer works it out; set it explicitly only to force the issue.

How It Works

A bias is redundant when the layer’s output is immediately normalized by something that applies its own learnable shift. The normalization subtracts the mean, which removes whatever constant the bias added, then adds its own beta in its place. Two parameters then describe one degree of freedom: the bias cannot receive a meaningful gradient, and it costs memory, checkpoint size and optimizer state for nothing.

Reference implementations usually hard-code this decision against the norm type they happen to use. pix2pix (Isola et al., 2017) writes use_bias = norm_layer == nn.InstanceNorm2d, which is right only because their InstanceNorm is built with affine=False and their BatchNorm with affine=True. The same expression is wrong for InstanceNorm2d(affine=True), which does supply a shift, and wrong for a scale-only normalization such as RMSNorm, which does not. Auto asks the following layer instead of inferring from its type, so it stays correct for normalizations those implementations never considered.

Fields

Field Summary
BiasMode Always Always carry a bias, even under a normalization that already shifts.
BiasMode Auto Decide from what follows: no bias when the next layer supplies a learnable shift, a bias otherwise.
BiasMode Never Never carry a bias.
BiasMode Unspecified Not stated.
int value__