HybridShardedOptimizer<T, TInput, TOutput>
Models & Types ยท AiDotNet.DistributedTraining
Implements 3D Parallelism optimizer - coordinates across data, tensor, and pipeline dimensions.
Inheritance: ShardedOptimizerBase<T, TInput, TOutput> โ HybridShardedOptimizer<T, TInput, TOutput>
For Beginners
This is the most complex optimizer, coordinating all three types of parallelism. It needs to handle:
- Averaging gradients across data-parallel replicas (GPUs processing different batches)
- Synchronizing tensor-parallel groups (GPUs sharing layer computations)
- Accumulating gradients from pipeline micro-batches
Think of it like coordinating a massive team split into departments (pipeline stages), work groups (tensor parallel), and shifts (data parallel) - all need to sync at the right times.
How It Works
Strategy Overview: 3D Parallelism optimizer coordinates optimization across all three parallelism dimensions:
- Data parallel: synchronizes gradients across data-parallel replicas
- Tensor parallel: synchronizes within tensor-parallel groups
- Pipeline parallel: handles gradient accumulation across micro-batches
This requires managing separate communication groups for each dimension and ensuring proper synchronization order to maintain correctness and efficiency.
Use Cases:
- Frontier-scale models (100B+ parameters)
- 100s to 1000s of GPUs
- Works with HybridShardedModel
Trade-offs:
- Memory: Excellent - exploits all dimensions
- Communication: Complex - multiple sync patterns
- Complexity: Very High - most complex optimizer
- Best for: Largest scale training
Constructors
| Constructor | Summary |
|---|---|
HybridShardedOptimizer(IOptimizer<T, TInput, TOutput> wrappedOptimizer, IShardingConfiguration<T> config, int pipelineParallelSize, int tensorParallelSize, int dataParallelSize) |
Methods
| Method | Summary |
|---|---|
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData) |
|
byte[] Serialize() |
|
void Deserialize(byte[] data) |
|
void SynchronizeOptimizerState() |