GhostBatchNormalization<T>
Models & Types · AiDotNet.NeuralNetworks.Tabular
Implements Ghost Batch Normalization, a regularization technique used in TabNet that applies batch normalization to virtual mini-batches within each actual batch.
Inheritance: LayerBase<T> → GhostBatchNormalization<T>
Implements: IShapeContract
For Beginners
Batch Normalization helps neural networks train faster by normalizing the inputs to each layer. Ghost Batch Normalization takes this further by adding controlled randomness through virtual batches.
Imagine you have a batch of 256 samples:
- Standard Batch Norm: Computes mean/variance over all 256 samples
- Ghost Batch Norm (virtual size 64): Computes 4 separate mean/variance calculations,
one for each group of 64 samples
This variation in statistics acts as regularization, helping prevent overfitting. It’s particularly effective for tabular data where overfitting is common.
How It Works
Ghost Batch Normalization divides each training batch into smaller “virtual batches” and computes separate normalization statistics for each. This provides a regularization effect similar to using smaller batch sizes without the computational overhead.
The scale (gamma) and shift (beta) are trainable parameters registered with the layer machinery, so a composite that registers this layer as a sub-layer exposes them to the parameter vector, checkpoints and the autodiff tape. The running statistics are non-trainable buffers.
Reference: “TabNet: Attentive Interpretable Tabular Learning” (Arik & Pfister, AAAI 2021)
Constructors
| Constructor | Summary |
|---|---|
GhostBatchNormalization(int numFeatures, int virtualBatchSize, double momentum, double epsilon) |
Initializes a new instance of the GhostBatchNormalization class. |
Properties
| Property | Summary |
|---|---|
Vector<T> Beta { get; } |
Gets a copy of the shift (beta) parameters. |
Vector<T> Gamma { get; } |
Gets a copy of the scale (gamma) parameters. |
Vector<T> RunningMean { get; } |
Gets a copy of the running mean statistics. |
Vector<T> RunningVar { get; } |
Gets a copy of the running variance statistics. |
bool SupportsTraining { get; } |
Gets whether this layer supports training. |
string Name { get; } |
Gets the name of this layer. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
Derived from this type’s [TensorLayout] declarations. |
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
IReadOnlyList<ValueTuple<string, Tensor<T>>> GetRegisteredBuffers() |
|
Tensor<T> ForwardInference(Tensor<T> input) |
Performs the forward pass using running statistics (inference mode). |
int[] GetOutputShape(int[] inputShape) |
Gets the output shape given an input shape. |
void ResetGradients() |
Clears the gradients accumulated for gamma and beta. |
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |