BatchEnsembleLayer<T>
Layers · AiDotNet.NeuralNetworks.Layers
Implements a BatchEnsemble layer that provides parameter-efficient ensembling.
Inheritance: LayerBase<T> → BatchEnsembleLayer<T>
Implements: IShapeContract
For Beginners
BatchEnsemble is a clever way to create multiple models (ensemble members) that share most of their weights.
Traditional ensemble: Train N separate models with N×parameters BatchEnsemble: Train 1 base model + N small vectors = ~1×parameters + small overhead
How it works:
- A single shared weight matrix W captures the main learned patterns
- Each ensemble member has two small vectors (r and s)
- Member i’s effective weights = W × (r_i outer-product s_i)
- This modulates the shared weights to create diversity
Benefits:
- Ensemble predictions (averaging multiple members) are usually more accurate
- Parameter cost is only slightly more than a single model
- Can process all members in parallel using batch operations
- Easy to implement and train
Example with 256-dim hidden layer and 4 members:
- Shared weights: 256 × 256 = 65,536 parameters
- Per-member vectors: 4 × (256 + 256) = 2,048 parameters
- Total overhead: ~3% more parameters for 4× ensemble benefit
How It Works
BatchEnsemble creates multiple ensemble members that share base weights but have their own small rank-1 matrices. For a weight matrix W, each member i computes: W_i = W ⊙ (r_i ⊗ s_i) where r_i and s_i are per-member rank vectors, ⊙ is element-wise multiplication, and ⊗ is outer product.
Constructors
| Constructor | Summary |
|---|---|
BatchEnsembleLayer(int inputDim, int outputDim, int numMembers, bool useBias, double rankInitScale) |
Initializes a new instance of the BatchEnsembleLayer class. |
Properties
| Property | Summary |
|---|---|
bool SupportsTraining { get; } |
|
int InputDim { get; } |
Gets the input dimension. |
int NumMembers { get; } |
Gets the number of ensemble members. |
int OutputDim { get; } |
Gets the output dimension. |
Methods
| Method | Summary |
|---|---|
Dictionary<string, string> GetParameterRoles() |
Returns parameter roles for per-role optimizer configuration (e.g., weight decay exemption for biases). |
IReadOnlyList<OutputAxisContract> OutputAxesFor(int inputRank) |
|
IReadOnlyList<Tensor<T>> GetTrainableParameters() |
Returns all trainable parameter tensors marked with [TrainableParameter]. |
Tensor<T> AverageMembers(Tensor<T> output) |
Averages the outputs across ensemble members. |
Tensor<T> ForwardExpanded(Tensor<T> expandedInput) |
Forward pass for input that is ALREADY expanded along the member axis ([batchSize * numMembers, inputDim], members in consecutive rows). |
Tensor<T> GetRVectors() |
Gets the r vectors tensor. |
Tensor<T> GetSVectors() |
Gets the s vectors tensor. |
Tensor<T> GetWeights() |
Gets the shared weights tensor. |
void ResetGradients() |
Resets all gradients. |
void ResetState() |
|
void SetTrainableParameters(IReadOnlyList<Tensor<T>> parameters) |
Replaces trainable parameter tensors (e.g., with ParameterBuffer views). |
void UpdateParameters(T learningRate) |
|
void ZeroGrad() |
Clears all gradient fields discovered by convention ({paramName}Gradient). |