LocalSGDOptimizer<T, TInput, TOutput>
Models & Types · AiDotNet.DistributedTraining
Implements Local SGD distributed training optimizer - parameter averaging after local optimization.
Inheritance: ShardedOptimizerBase<T, TInput, TOutput> → LocalSGDOptimizer<T, TInput, TOutput>
For Beginners
Unlike traditional DDP which synchronizes gradients before every parameter update, Local SGD lets each worker train independently for several steps, then averages the final model parameters. Think of it like students studying independently for a week, then meeting to average their understanding, rather than checking answers after every practice problem.
How It Works
Strategy Overview: Local SGD allows each worker to perform multiple local optimization steps independently, then synchronizes model parameters (not gradients) across all workers using AllReduce averaging. This reduces communication frequency compared to traditional DDP while maintaining convergence. Based on “Don’t Use Large Mini-Batches, Use Local SGD” (Lin et al., 2020).
Key Difference from DDP:
- Local SGD (this class): Optimize locally → Average PARAMETERS → Continue training
- True DDP: Compute gradients → Average GRADIENTS → Apply averaged gradients → Continue training
Use Cases:
- Reducing communication frequency in distributed training
- Slower network connections where communication is expensive
- Works with any optimizer (Adam, SGD, RMSprop, etc.)
- Large models where parameter synchronization dominates training time
Trade-offs:
- Memory: Each process stores full model and optimizer state
- Communication: Very low - parameters synchronized less frequently than gradients
- Convergence: Slightly different trajectory than DDP but reaches similar final accuracy
- Complexity: Low - straightforward parameter averaging
- Best for: Communication-constrained distributed training
Production Note: For true DDP (gradient averaging), use GradientCompressionOptimizer with compression ratio = 1.0, which properly averages gradients before parameter updates.
Constructors
| Constructor | Summary |
|---|---|
LocalSGDOptimizer(IOptimizer<T, TInput, TOutput> wrappedOptimizer, IShardingConfiguration<T> config) |
Creates a Local SGD optimizer that averages parameters across workers. |
Methods
| Method | Summary |
|---|---|
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData) |
|
byte[] Serialize() |
|
void Deserialize(byte[] data) |
|
void SynchronizeOptimizerState() |