IShardedOptimizer<T, TInput, TOutput>

Interfaces · AiDotNet.DistributedTraining

Defines the contract for optimizers that support distributed training with parameter sharding.

Implements: IOptimizer<T, TInput, TOutput>

Known implementations: ShardedOptimizerBase<T, TInput, TOutput>

For Beginners

A sharded optimizer is like having a team of coaches working together. Each coach (process) is responsible for updating a portion of the player’s (model’s) skills. After each round of practice, the coaches share and combine their improvements to ensure everyone stays in sync.

How It Works

This allows optimizing very large models that don’t fit on a single GPU.

Properties

Property Summary
IOptimizer<T, TInput, TOutput> WrappedOptimizer { get; } Gets the underlying wrapped optimizer.
IShardingConfiguration<T> ShardingConfiguration { get; } Gets the sharding configuration for this optimizer.
int Rank { get; } Gets the rank of this process in the distributed group.
int WorldSize { get; } Gets the total number of processes in the distributed group.

Methods

Method Summary
void SynchronizeOptimizerState() Synchronizes optimizer state (like momentum buffers) across all processes.