IShardedOptimizer<T, TInput, TOutput>
Interfaces · AiDotNet.DistributedTraining
Defines the contract for optimizers that support distributed training with parameter sharding.
Implements: IOptimizer<T, TInput, TOutput>
Known implementations: ShardedOptimizerBase<T, TInput, TOutput>
For Beginners
A sharded optimizer is like having a team of coaches working together. Each coach (process) is responsible for updating a portion of the player’s (model’s) skills. After each round of practice, the coaches share and combine their improvements to ensure everyone stays in sync.
How It Works
This allows optimizing very large models that don’t fit on a single GPU.
Properties
| Property | Summary |
|---|---|
IOptimizer<T, TInput, TOutput> WrappedOptimizer { get; } |
Gets the underlying wrapped optimizer. |
IShardingConfiguration<T> ShardingConfiguration { get; } |
Gets the sharding configuration for this optimizer. |
int Rank { get; } |
Gets the rank of this process in the distributed group. |
int WorldSize { get; } |
Gets the total number of processes in the distributed group. |
Methods
| Method | Summary |
|---|---|
void SynchronizeOptimizerState() |
Synchronizes optimizer state (like momentum buffers) across all processes. |