GradientCompressionOptimizer<T, TInput, TOutput>
Models & Types · AiDotNet.DistributedTraining
Implements gradient compression optimizer - wraps any optimizer to add gradient compression.
Inheritance: ShardedOptimizerBase<T, TInput, TOutput> → GradientCompressionOptimizer<T, TInput, TOutput>
For Beginners
This optimizer wrapper compresses gradients before sending them across the network. Think of it like compressing a large file before emailing it - the information is mostly preserved, but it takes much less time to transmit. This is especially useful when your GPUs are connected by slower networks.
Common compression techniques:
- Make numbers less precise (like rounding 3.14159 to 3.14)
- Send only the most important values (top-k sparsification)
- Use mathematical tricks to represent data more compactly
How It Works
Strategy Overview: Gradient compression reduces communication overhead in distributed training by compressing gradients before transmission. Common techniques include:
- Quantization: Reduce precision (FP32 → FP16 or INT8)
- Sparsification: Send only top-k largest gradients
- Low-rank compression: Decompose gradient matrices
This can significantly reduce network bandwidth usage (2-100x compression) with minimal accuracy impact when done carefully. Essential for training over slow networks or at very large scales.
Use Cases:
- Slow network connections (cross-datacenter, ethernet vs NVLink)
- Very large models where gradient size is huge
- Can wrap any other optimizer (DDP, FSDP, etc.)
- Bandwidth-constrained scenarios
Trade-offs:
- Memory: Same as wrapped optimizer
- Communication: Much lower - 2x to 100x reduction depending on technique
- Complexity: Moderate - adds compression/decompression overhead
- Accuracy: Slight potential degradation (usually <1% with proper tuning)
- Best for: Slow networks, bandwidth-limited scenarios
Implementation Note: This framework provides the compression optimizer infrastructure. Specific compression algorithms (quantization levels, sparsity ratios) can be configured. Production use would implement specific compression strategies based on your network and accuracy requirements.
Example:
Constructors
| Constructor | Summary |
|---|---|
GradientCompressionOptimizer(IOptimizer<T, TInput, TOutput> wrappedOptimizer, IShardingConfiguration<T> config, double compressionRatio, bool useQuantization, bool useSparsification) |
Creates a gradient compression optimizer. |
Methods
| Method | Summary |
|---|---|
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData) |
|
byte[] Serialize() |
|
void Deserialize(byte[] data) |
|
void SynchronizeOptimizerState() |