AsyncSGDOptimizer<T, TInput, TOutput>
Models & Types · AiDotNet.DistributedTraining
Implements Asynchronous SGD optimizer - allows asynchronous parameter updates without strict barriers.
Inheritance: ShardedOptimizerBase<T, TInput, TOutput> → AsyncSGDOptimizer<T, TInput, TOutput>
For Beginners
Async SGD is like a team working independently without meetings. Each person:
- Reads current parameters
- Computes gradients on their data
- Updates parameters immediately (no waiting!)
Pro: No time wasted waiting for slow workers Con: Updates might conflict or use slightly stale information
This works well when updates are sparse (touching different parameters) but can be unstable when all workers update the same parameters frequently.
How It Works
Strategy Overview: Asynchronous SGD (and variants like Hogwild!) removes synchronization barriers between workers. Each process updates parameters independently without waiting for others, using a parameter server or shared memory. This eliminates idle time from synchronization but introduces stale gradients - workers may compute gradients on slightly outdated parameters.
When done correctly (sparse gradients, low contention), async SGD can achieve near-linear speedup without much accuracy loss. However, it’s more sensitive to hyperparameters and can be unstable for dense updates.
Use Cases:
- Sparse models (embeddings, recommendation systems)
- Scenarios with stragglers (some workers slower than others)
- When synchronization overhead is very high
- Research and experimentation
Trade-offs:
- Memory: Requires parameter server or shared memory
- Communication: Asynchronous - can be higher total volume
- Complexity: High - requires parameter server infrastructure
- Convergence: Can be slower or less stable than sync SGD
- Best for: Sparse updates, heterogeneous workers, straggler tolerance
- Limitation: Harder to tune, may require staleness-aware algorithms
Implementation Note: This framework provides async SGD infrastructure. Full production implementation requires parameter server setup or shared memory coordination. This implementation demonstrates the async update pattern.
Example:
Constructors
| Constructor | Summary |
|---|---|
AsyncSGDOptimizer(IOptimizer<T, TInput, TOutput> wrappedOptimizer, IShardingConfiguration<T> config, int allowStaleness) |
Creates an async SGD optimizer. |
Properties
| Property | Summary |
|---|---|
int MaxStaleness { get; } |
Maximum number of steps a worker may run ahead of the slowest worker before it must synchronize (the Stale-Synchronous Parallel staleness bound, Ho et al. |
Methods
| Method | Summary |
|---|---|
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData) |
|
bool ShouldSync(int iteration) |
Checks if a barrier should be used (for periodic synchronization). |
byte[] Serialize() |
|
void Deserialize(byte[] data) |
|
void SynchronizeOptimizerState() |