ElasticOptimizer<T, TInput, TOutput>
Models & Types · AiDotNet.DistributedTraining
Implements Elastic optimizer - supports dynamic worker addition/removal during training.
Inheritance: ShardedOptimizerBase<T, TInput, TOutput> → ElasticOptimizer<T, TInput, TOutput>
For Beginners
Elastic training is like having a flexible team size. Workers can join or leave during training without stopping everything:
Scenario 1 - Fault tolerance:
- Start with 8 GPUs training your model
- GPU 3 fails → automatically detected
- Training continues with 7 GPUs (parameters redistributed)
- New GPU joins → training scales back to 8 GPUs
Scenario 2 - Cloud cost optimization:
- Use cheap “spot instances” that can be taken away anytime
- When instance is preempted, training continues with remaining workers
- New instance joins when available
This is critical for long training jobs where failures are expected.
How It Works
Strategy Overview: Elastic training (TorchElastic, Horovod Elastic) enables dynamic scaling of workers during training. Workers can be added or removed without stopping the training job, supporting:
- Fault tolerance: Replace failed workers automatically
- Auto-scaling: Add workers during peak hours, remove during off-peak
- Spot instance usage: Tolerate preemptions, use cheaper compute
When world size changes, the optimizer handles re-sharding parameters and optimizer states across the new worker set. This requires checkpointing and careful state management.
Use Cases:
- Long training jobs (days/weeks) where failures will occur
- Cloud training with spot/preemptible instances (save 60-90% cost)
- Auto-scaling based on load or time of day
- Fault tolerance for production training pipelines
Trade-offs:
- Memory: Must handle dynamic re-sharding
- Communication: Overhead during worker changes (re-sharding, sync)
- Complexity: Very High - requires membership management, state re-distribution
- Convergence: Learning rate scheduling must account for dynamic world size
- Best for: Long jobs, cost-sensitive scenarios, production ML pipelines
- Limitation: Worker changes create temporary slowdown during re-sharding
Implementation Note: This framework provides elastic optimizer infrastructure. Full production deployment requires:
- Membership/discovery service (etcd, ZooKeeper, or cloud-native)
- Automatic checkpointing before worker changes
- State re-sharding algorithms
- Rendezvous mechanism for worker coordination
This implementation demonstrates the elastic pattern.
Example:
Constructors
| Constructor | Summary |
|---|---|
ElasticOptimizer(IOptimizer<T, TInput, TOutput> wrappedOptimizer, IShardingConfiguration<T> config, int minWorkers, int maxWorkers) |
Creates an elastic optimizer. |
Properties
| Property | Summary |
|---|---|
bool CanScaleDown { get; } |
Gets whether the optimizer can tolerate losing workers. |
bool CanScaleUp { get; } |
Gets whether the optimizer can accept more workers. |
int CurrentWorkers { get; } |
Gets the current number of active workers. |
Methods
| Method | Summary |
|---|---|
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData) |
|
byte[] Serialize() |
|
void Deserialize(byte[] data) |
|
void SynchronizeOptimizerState() |