ZeRO1Optimizer<T, TInput, TOutput>

Models & Types · AiDotNet.DistributedTraining

Implements ZeRO Stage 1 optimizer - shards optimizer states only.

Inheritance: ShardedOptimizerBase<T, TInput, TOutput> → ZeRO1Optimizer<T, TInput, TOutput>

For Beginners

This optimizer saves memory by splitting the optimizer’s internal memory (like momentum in Adam) across processes. The model parameters are still fully replicated, but each process only stores a portion of the optimizer’s “memory” or “state”. When it’s time to update parameters, processes share their pieces of the optimizer state as needed.

How It Works

Strategy Overview: ZeRO-1 optimizer shards optimizer states (momentum buffers, variance estimates) across processes while keeping parameters and gradients replicated. This reduces memory overhead from optimizer state (which can be 4x the model size for Adam: fp32 params + momentum + variance + gradients). When needed for updates, optimizer states are gathered from their respective owners.

Use Cases:

  • Using stateful optimizers (Adam, RMSprop) with limited memory
  • Want memory savings without full ZeRO-3/FSDP complexity
  • Works well with ZeRO1Model

Trade-offs:

  • Memory: Good - saves ~4x memory from optimizer states for Adam
  • Communication: Low - same as DDP plus occasional state gather
  • Complexity: Moderate - state sharding adds some complexity
  • Best for: Memory-constrained scenarios with stateful optimizers

Constructors

Constructor Summary
ZeRO1Optimizer(IOptimizer<T, TInput, TOutput> wrappedOptimizer, IShardingConfiguration<T> config)

Methods

Method Summary
OptimizationResult<T, TInput, TOutput> Optimize(OptimizationInputData<T, TInput, TOutput> inputData)
byte[] Serialize()
void Deserialize(byte[] data)
void SynchronizeOptimizerState()