InferenceFusionSettings

Options & Configuration · AiDotNet.NeuralNetworks

Inference-time graph fusions applied by the layers themselves.

How It Works

Batch-norm folding. In inference a batch normalization after a convolution is a fixed per-channel affine map, so it is folded into the convolution: W' = W * gamma / sqrt(var + eps) and b' = (b - mean) * gamma / sqrt(var + eps) + beta, and a ReLU that follows runs in the same kernel. This is what PyTorch’s frozen models (torch.jit.freeze), ONNX export and TensorRT do. It saves the batch norm’s pass over every activation (and the ReLU’s), at the cost of a different rounding: outputs differ from the unfused convolution, batch norm and ReLU by about 1e-6 relative in float. Training, a recording gradient tape and graph capture never fold.

Default ON. Set AIDOTNET_FOLD_BATCHNORM=0, or FoldBatchNormIntoConvolution to false, to run the layers unfused (bit-for-bit the separate operations).

Properties

Property Summary
bool FoldBatchNormIntoConvolution { get; set; } Whether inference folds a batch normalization (and a ReLU after it) into the preceding convolution.