InferenceFusionSettings
Options & Configuration · AiDotNet.NeuralNetworks
Inference-time graph fusions applied by the layers themselves.
How It Works
Batch-norm folding. In inference a batch normalization after a convolution is a fixed per-channel affine map,
so it is folded into the convolution: W' = W * gamma / sqrt(var + eps) and
b' = (b - mean) * gamma / sqrt(var + eps) + beta, and a ReLU that follows runs in the same kernel. This is
what PyTorch’s frozen models (torch.jit.freeze), ONNX export and TensorRT do. It saves the batch norm’s pass
over every activation (and the ReLU’s), at the cost of a different rounding: outputs differ from the unfused
convolution, batch norm and ReLU by about 1e-6 relative in float. Training, a recording gradient tape and graph
capture never fold.
Default ON. Set AIDOTNET_FOLD_BATCHNORM=0, or FoldBatchNormIntoConvolution to
false, to run the layers unfused (bit-for-bit the separate operations).
Properties
| Property | Summary |
|---|---|
bool FoldBatchNormIntoConvolution { get; set; } |
Whether inference folds a batch normalization (and a ReLU after it) into the preceding convolution. |