ZeRO3Optimizer<T, TInput, TOutput>

Models & Types · AiDotNet.DistributedTraining

Implements ZeRO Stage 3 optimizer - full sharding equivalent to FSDP.

Inheritance: ShardedOptimizerBase<T, TInput, TOutput> → FSDPOptimizer<T, TInput, TOutput> → ZeRO3Optimizer<T, TInput, TOutput>

For Beginners

ZeRO-3 and FSDP optimizers are the same thing. Use whichever name you prefer. Everything is sharded for maximum memory efficiency.

How It Works

Strategy Overview: ZeRO-3 is equivalent to FSDP optimizer - full sharding of parameters, gradients, and optimizer states. This class is an alias to FSDPOptimizer for ZeRO terminology consistency.

Constructors

Constructor Summary
ZeRO3Optimizer(IOptimizer<T, TInput, TOutput> wrappedOptimizer, IShardingConfiguration<T> config)