FSDPOptimizer<T, TInput, TOutput>

Models & Types ยท AiDotNet.DistributedTraining

Implements FSDP (Fully Sharded Data Parallel) optimizer wrapper that coordinates optimization across multiple processes.

Inheritance: ShardedOptimizerBase<T, TInput, TOutput> โ†’ FSDPOptimizer<T, TInput, TOutput>

Known derived types: ZeRO3Optimizer<T, TInput, TOutput>

For Beginners

This class wraps any existing optimizer (like Adam, SGD, etc.) and makes it work with FSDP strategy across multiple GPUs or machines. It automatically handles:

  • Synchronizing gradients across all processes
  • Sharding optimizer states (momentum, variance) to save memory
  • Coordinating parameter updates
  • Ensuring all processes stay in sync

How It Works

Strategy Overview: FSDP optimizer works in conjunction with FSDPModel to provide full sharding of optimizer states. This means momentum buffers, variance estimates, and all other optimizer-specific state are sharded across processes, minimizing memory usage while maintaining training effectiveness.

Think of it like a team of coaches working together - each coach has their own expertise (the wrapped optimizer), but they share only the essential information and keep their detailed notes (optimizer states) private to save space.

Use Cases:

  • Training very large models with optimizers that have significant state (Adam, RMSprop)
  • Maximizing memory efficiency when using stateful optimizers
  • Scaling to hundreds or thousands of GPUs

Trade-offs:

  • Memory: Excellent - shards optimizer states across processes
  • Communication: Moderate - syncs gradients and occasional state synchronization
  • Complexity: Moderate - automatic state sharding
  • Best for: Large models with stateful optimizers (Adam, RMSprop, etc.)

Example:

Constructors

Constructor Summary
FSDPOptimizer(IOptimizer<T, TInput, TOutput> wrappedOptimizer, IShardingConfiguration<T> config) Creates a new FSDP optimizer wrapping an existing optimizer.

Methods

Method Summary
OptimizationAlgorithmOptions<T, TInput, TOutput> GetOptions()
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData)
bool ShouldEarlyStop()
byte[] Serialize()
void Deserialize(byte[] data)
void SynchronizeOptimizerState()