BranchformerBlock<T>

Layers · AiDotNet.NeuralNetworks.Layers

A Branchformer encoder block per Peng et al., ICML 2022, “Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understanding” (arXiv:2207.02971).

Inheritance: LayerBase<T> → BranchformerBlock<T>

Implements: IShapeContract

How It Works

Two branches run in PARALLEL over the same input: self-attention captures long-range dependencies, and a convolutional gating MLP (cgMLP) captures local ones. Their outputs are concatenated and projected back to the model width.

This is the distinction from Conformer, which composes attention and convolution SEQUENTIALLY inside one branch. The parallel arrangement is what lets the model weigh the two context types against each other per layer, and it is the paper’s entire contribution — a Branchformer built out of Conformer blocks is just a Conformer.

cgMLP and the CSGU: a channel projection with GeLU expands to the hidden width, then the Convolutional Spatial Gating Unit splits that in half, applies layer norm and a depth-wise convolution to one half, and multiplies the two halves element-wise. The paper notes the gating is LINEAR — no extra nonlinearity between the projection and the multiplication — so the depth-wise path acts as a learned spatial gate rather than another activation.

Why a composite layer rather than loose layers: a parallel structure flattened into a sequential layer list would be executed as attention-then-cgMLP by any default forward pass, silently turning it back into a sequential block. Keeping both branches inside one layer makes that impossible.

Gradient tracking: every operation goes through IEngine, so the tape records both branches and the merge without a manual backward.

Constructors

Constructor Summary
BranchformerBlock(int modelDim, int numHeads, int cgmlpHiddenDim, int kernelSize) Initializes a new Branchformer block.

Properties

Property Summary
bool SupportsTraining { get; }

Methods

Method Summary
IReadOnlyList<ILayer<T>> GetSubLayers()
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) Derived from this type’s [TensorLayout] declarations.
void ResetState()