FSDPOptimizer<T, TInput, TOutput>
Models & Types ยท AiDotNet.DistributedTraining
Implements FSDP (Fully Sharded Data Parallel) optimizer wrapper that coordinates optimization across multiple processes.
Inheritance: ShardedOptimizerBase<T, TInput, TOutput> โ FSDPOptimizer<T, TInput, TOutput>
Known derived types: ZeRO3Optimizer<T, TInput, TOutput>
For Beginners
This class wraps any existing optimizer (like Adam, SGD, etc.) and makes it work with FSDP strategy across multiple GPUs or machines. It automatically handles:
- Synchronizing gradients across all processes
- Sharding optimizer states (momentum, variance) to save memory
- Coordinating parameter updates
- Ensuring all processes stay in sync
How It Works
Strategy Overview: FSDP optimizer works in conjunction with FSDPModel to provide full sharding of optimizer states. This means momentum buffers, variance estimates, and all other optimizer-specific state are sharded across processes, minimizing memory usage while maintaining training effectiveness.
Think of it like a team of coaches working together - each coach has their own expertise (the wrapped optimizer), but they share only the essential information and keep their detailed notes (optimizer states) private to save space.
Use Cases:
- Training very large models with optimizers that have significant state (Adam, RMSprop)
- Maximizing memory efficiency when using stateful optimizers
- Scaling to hundreds or thousands of GPUs
Trade-offs:
- Memory: Excellent - shards optimizer states across processes
- Communication: Moderate - syncs gradients and occasional state synchronization
- Complexity: Moderate - automatic state sharding
- Best for: Large models with stateful optimizers (Adam, RMSprop, etc.)
Example:
Constructors
| Constructor | Summary |
|---|---|
FSDPOptimizer(IOptimizer<T, TInput, TOutput> wrappedOptimizer, IShardingConfiguration<T> config) |
Creates a new FSDP optimizer wrapping an existing optimizer. |
Methods
| Method | Summary |
|---|---|
OptimizationAlgorithmOptions<T, TInput, TOutput> GetOptions() |
|
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData) |
|
bool ShouldEarlyStop() |
|
byte[] Serialize() |
|
void Deserialize(byte[] data) |
|
void SynchronizeOptimizerState() |