LoadBalancedPartitionStrategy<T>

Models & Types · AiDotNet.DistributedTraining

Partitions model parameters across pipeline stages using estimated computational cost per layer.

Implements: IPipelinePartitionStrategy<T>

For Beginners

Imagine an assembly line where some tasks take much longer than others. If you assign tasks purely by count, some workers finish early and wait while others are still busy. This strategy assigns tasks by estimated time, so all workers finish at roughly the same time.

For neural networks, attention layers are much more expensive than simple normalization layers, so this strategy gives fewer attention layers to each stage to balance the workload.

The cost function estimates FLOPs (floating point operations) for a block of parameters:

  • Dense/linear layers: ~2 * inputSize * outputSize FLOPs
  • Attention: ~4 * seqLen * d_model FLOPs
  • LayerNorm: ~5 * d_model FLOPs

Since we don’t have layer-level metadata in the parameter vector, costs are estimated from parameter counts using the heuristic that computation scales quadratically with matrix dimensions.

How It Works

Instead of dividing parameters uniformly, this strategy uses a cost function to estimate the computational load for each parameter group (layer). It then assigns parameters to stages so that each stage has roughly equal total cost, reducing pipeline bubble overhead.

Reference: Megatron-LM layer assignment algorithm, NVIDIA 2020.

Constructors

Constructor Summary
LoadBalancedPartitionStrategy(int estimatedLayerSize, Func<int, T> costEstimator) Creates a load-balanced partition strategy that auto-detects layer boundaries using a fixed layer size estimate.
LoadBalancedPartitionStrategy(int[] layerBoundaries, Func<int, T> costEstimator) Creates a load-balanced partition strategy that auto-detects layer boundaries using a fixed layer size estimate.

Methods

Method Summary
ValueTuple<int, int>[] ComputePartition(int totalParameters, int numStages)