ZeRO3Model<T, TInput, TOutput>

Models & Types · AiDotNet.DistributedTraining

Implements ZeRO Stage 3 model wrapper - full sharding of parameters, gradients, and optimizer states.

Inheritance: ShardedModelBase<T, TInput, TOutput> → FSDPModel<T, TInput, TOutput> → ZeRO3Model<T, TInput, TOutput>

For Beginners

ZeRO-3 is identical to FSDP - it’s the ultimate memory-saving strategy. Everything is sharded: parameters, gradients, and optimizer states. Each process only holds a small piece of the model, and pieces are gathered only when absolutely needed, then immediately released.

How It Works

Strategy Overview: ZeRO Stage 3 is the full implementation of the ZeRO optimization, sharding parameters, gradients, AND optimizer states across all processes. This is equivalent to PyTorch’s FSDP (Fully Sharded Data Parallel). Parameters are gathered just-in-time for forward/backward passes and immediately released, maximizing memory efficiency.

This class is essentially an alias/wrapper for FSDPModel to maintain ZeRO naming consistency.

Use Cases:

  • Same as FSDP - training very large models
  • When you prefer ZeRO terminology over FSDP
  • Maximum memory efficiency

Trade-offs:

  • Same as FSDP
  • Memory: Excellent - everything sharded
  • Communication: Higher - AllGather for each forward/backward
  • Complexity: Moderate

Example:

Constructors

Constructor Summary
ZeRO3Model(IFullModel<T, TInput, TOutput> wrappedModel, IShardingConfiguration<T> config) Creates a new ZeRO-3 model wrapping an existing model.

Methods

Method Summary
IFullModel<T, TInput, TOutput> WithParameters(Vector<T> parameters)
ModelMetadata<T> GetModelMetadata()