HybridShardedOptimizer<T, TInput, TOutput>

Models & Types ยท AiDotNet.DistributedTraining

Implements 3D Parallelism optimizer - coordinates across data, tensor, and pipeline dimensions.

Inheritance: ShardedOptimizerBase<T, TInput, TOutput> โ†’ HybridShardedOptimizer<T, TInput, TOutput>

For Beginners

This is the most complex optimizer, coordinating all three types of parallelism. It needs to handle:

  1. Averaging gradients across data-parallel replicas (GPUs processing different batches)
  2. Synchronizing tensor-parallel groups (GPUs sharing layer computations)
  3. Accumulating gradients from pipeline micro-batches

Think of it like coordinating a massive team split into departments (pipeline stages), work groups (tensor parallel), and shifts (data parallel) - all need to sync at the right times.

How It Works

Strategy Overview: 3D Parallelism optimizer coordinates optimization across all three parallelism dimensions:

  • Data parallel: synchronizes gradients across data-parallel replicas
  • Tensor parallel: synchronizes within tensor-parallel groups
  • Pipeline parallel: handles gradient accumulation across micro-batches

This requires managing separate communication groups for each dimension and ensuring proper synchronization order to maintain correctness and efficiency.

Use Cases:

  • Frontier-scale models (100B+ parameters)
  • 100s to 1000s of GPUs
  • Works with HybridShardedModel

Trade-offs:

  • Memory: Excellent - exploits all dimensions
  • Communication: Complex - multiple sync patterns
  • Complexity: Very High - most complex optimizer
  • Best for: Largest scale training

Constructors

Constructor Summary
HybridShardedOptimizer(IOptimizer<T, TInput, TOutput> wrappedOptimizer, IShardingConfiguration<T> config, int pipelineParallelSize, int tensorParallelSize, int dataParallelSize)

Methods

Method Summary
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData)
byte[] Serialize()
void Deserialize(byte[] data)
void SynchronizeOptimizerState()