ShardedOptimizerBase<T, TInput, TOutput>
Base Classes · AiDotNet.DistributedTraining
Provides base implementation for distributed optimizers with parameter sharding.
Implements: IModelShape, IShardedOptimizer<T, TInput, TOutput>
Known derived types: AsyncSGDOptimizer<T, TInput, TOutput>, DDPOptimizer<T, TInput, TOutput>, ElasticOptimizer<T, TInput, TOutput>, FSDPOptimizer<T, TInput, TOutput>, GradientCompressionOptimizer<T, TInput, TOutput>, HybridShardedOptimizer<T, TInput, TOutput>, LocalSGDOptimizer<T, TInput, TOutput>, PipelineParallelOptimizer<T, TInput, TOutput>, TensorParallelOptimizer<T, TInput, TOutput>, ZeRO1Optimizer<T, TInput, TOutput>, ZeRO2Optimizer<T, TInput, TOutput>
For Beginners
This is the foundation that all distributed optimizers build upon.
Think of this as a template for coordinating optimization across multiple computers or GPUs. It handles common tasks like:
- Wrapping regular optimizers to work in distributed mode
- Syncing parameters across all processes after updates
- Making sure all processes agree on when to stop training
- Saving and loading distributed optimizer state
Specific types of distributed optimizers (like data-parallel or ZeRO) inherit from this and add their own strategies. This prevents code duplication and ensures all distributed optimizers work consistently.
How It Works
This abstract class implements common functionality for all sharded optimizers, including optimizer wrapping, parameter synchronization, consensus-based early stopping, and serialization. Derived classes can customize the optimization strategy, implement different sharding approaches (FSDP, ZeRO, etc.), or add optimizer-specific features.
Properties
| Property | Summary |
|---|---|
IOptimizer<T, TInput, TOutput> WrappedOptimizer { get; } |
|
IShardingConfiguration<T> ShardingConfiguration { get; } |
|
Vector<T> LastComputedGradients { get; } |
Gets the gradients computed during the last optimization step. |
int Rank { get; } |
|
int WorldSize { get; } |
Methods
| Method | Summary |
|---|---|
DynamicShapeInfo GetDynamicShapeInfo() |
|
IFullModel<T, TInput, TOutput> ApplyGradients(Vector<T> gradients, IFullModel<T, TInput, TOutput> model) |
Applies pre-computed gradients to a model’s parameters. |
OptimizationAlgorithmOptions<T, TInput, TOutput> GetOptions() |
|
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData) |
|
bool ShouldEarlyStop() |
|
byte[] Serialize() |
|
int[] GetInputShape() |
|
int[] GetOutputShape() |
|
void Deserialize(byte[] data) |
|
void LoadModel(string filePath) |
|
void Reset() |
|
void SaveModel(string filePath) |
|
void SetModel(IFullModel<T, TInput, TOutput> model) |
|
void SynchronizeOptimizerState() |