ShardedOptimizerBase<T, TInput, TOutput>

Base Classes · AiDotNet.DistributedTraining

Provides base implementation for distributed optimizers with parameter sharding.

Implements: IModelShape, IShardedOptimizer<T, TInput, TOutput>

Known derived types: AsyncSGDOptimizer<T, TInput, TOutput>, DDPOptimizer<T, TInput, TOutput>, ElasticOptimizer<T, TInput, TOutput>, FSDPOptimizer<T, TInput, TOutput>, GradientCompressionOptimizer<T, TInput, TOutput>, HybridShardedOptimizer<T, TInput, TOutput>, LocalSGDOptimizer<T, TInput, TOutput>, PipelineParallelOptimizer<T, TInput, TOutput>, TensorParallelOptimizer<T, TInput, TOutput>, ZeRO1Optimizer<T, TInput, TOutput>, ZeRO2Optimizer<T, TInput, TOutput>

For Beginners

This is the foundation that all distributed optimizers build upon.

Think of this as a template for coordinating optimization across multiple computers or GPUs. It handles common tasks like:

  • Wrapping regular optimizers to work in distributed mode
  • Syncing parameters across all processes after updates
  • Making sure all processes agree on when to stop training
  • Saving and loading distributed optimizer state

Specific types of distributed optimizers (like data-parallel or ZeRO) inherit from this and add their own strategies. This prevents code duplication and ensures all distributed optimizers work consistently.

How It Works

This abstract class implements common functionality for all sharded optimizers, including optimizer wrapping, parameter synchronization, consensus-based early stopping, and serialization. Derived classes can customize the optimization strategy, implement different sharding approaches (FSDP, ZeRO, etc.), or add optimizer-specific features.

Properties

Property Summary
IOptimizer<T, TInput, TOutput> WrappedOptimizer { get; }
IShardingConfiguration<T> ShardingConfiguration { get; }
Vector<T> LastComputedGradients { get; } Gets the gradients computed during the last optimization step.
int Rank { get; }
int WorldSize { get; }

Methods

Method Summary
DynamicShapeInfo GetDynamicShapeInfo()
IFullModel<T, TInput, TOutput> ApplyGradients(Vector<T> gradients, IFullModel<T, TInput, TOutput> model) Applies pre-computed gradients to a model’s parameters.
OptimizationAlgorithmOptions<T, TInput, TOutput> GetOptions()
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData)
bool ShouldEarlyStop()
byte[] Serialize()
int[] GetInputShape()
int[] GetOutputShape()
void Deserialize(byte[] data)
void LoadModel(string filePath)
void Reset()
void SaveModel(string filePath)
void SetModel(IFullModel<T, TInput, TOutput> model)
void SynchronizeOptimizerState()