BatchEnsembleLayer<T>

Layers · AiDotNet.NeuralNetworks.Layers

Implements a BatchEnsemble layer that provides parameter-efficient ensembling.

Inheritance: LayerBase<T> → BatchEnsembleLayer<T>

Implements: IShapeContract

For Beginners

BatchEnsemble is a clever way to create multiple models (ensemble members) that share most of their weights.

Traditional ensemble: Train N separate models with N×parameters BatchEnsemble: Train 1 base model + N small vectors = ~1×parameters + small overhead

How it works:

  1. A single shared weight matrix W captures the main learned patterns
  2. Each ensemble member has two small vectors (r and s)
  3. Member i’s effective weights = W × (r_i outer-product s_i)
  4. This modulates the shared weights to create diversity

Benefits:

  • Ensemble predictions (averaging multiple members) are usually more accurate
  • Parameter cost is only slightly more than a single model
  • Can process all members in parallel using batch operations
  • Easy to implement and train

Example with 256-dim hidden layer and 4 members:

  • Shared weights: 256 × 256 = 65,536 parameters
  • Per-member vectors: 4 × (256 + 256) = 2,048 parameters
  • Total overhead: ~3% more parameters for 4× ensemble benefit

How It Works

BatchEnsemble creates multiple ensemble members that share base weights but have their own small rank-1 matrices. For a weight matrix W, each member i computes: W_i = W ⊙ (r_i ⊗ s_i) where r_i and s_i are per-member rank vectors, ⊙ is element-wise multiplication, and ⊗ is outer product.

Constructors

Constructor Summary
BatchEnsembleLayer(int inputDim, int outputDim, int numMembers, bool useBias, double rankInitScale) Initializes a new instance of the BatchEnsembleLayer class.

Properties

Property Summary
bool SupportsTraining { get; }
int InputDim { get; } Gets the input dimension.
int NumMembers { get; } Gets the number of ensemble members.
int OutputDim { get; } Gets the output dimension.

Methods

Method Summary
Dictionary<string, string> GetParameterRoles() Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases).
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank)
IReadOnlyList<Tensor<T>> GetTrainableParameters() Returns all trainable parameter tensors marked with [TrainableParameter].
Tensor<T> AverageMembers(Tensor<T> output) Averages the outputs across ensemble members.
Tensor<T> ForwardExpanded(Tensor<T> expandedInput) Forward pass for input that is ALREADY expanded along the member axis ([batchSize * numMembers, inputDim], members in consecutive rows).
Tensor<T> GetRVectors() Gets the r vectors tensor.
Tensor<T> GetSVectors() Gets the s vectors tensor.
Tensor<T> GetWeights() Gets the shared weights tensor.
void ResetGradients() Resets all gradients.
void ResetState()
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) Replaces trainable parameter tensors (e.g., with ParameterBuffer views).
void UpdateParameters(T learningRate)
void ZeroGrad() Clears all gradient fields discovered by convention ({paramName}Gradient).