IShardingConfiguration<T>
Interfaces · AiDotNet.DistributedTraining
Configuration for parameter sharding in distributed training.
Known implementations: ShardingConfiguration<T>
For Beginners
This configuration tells the sharding system how to divide up parameters and how to handle communication. Think of it as the “rules” for how the team collaborates.
Properties
| Property | Summary |
|---|---|
ICommunicationBackend<T> CommunicationBackend { get; } |
Gets the communication backend to use for distributed operations. |
T LearningRate { get; } |
Gets the learning rate for gradient application during training. |
bool AutoSyncGradients { get; } |
Gets whether to automatically synchronize gradients after backward pass. |
bool CpuOffloadGradients { get; } |
Gets whether to page gradient tensors to CPU RAM before the sharded reduction (all-reduce for DDP, reduce-scatter for FSDP/ZeRO). |
bool CpuOffloadOptimizer { get; } |
Gets whether to keep optimizer state (Adam’s m/v moments, momentum buffers, etc.) on the CPU and run the optimizer step on the CPU too. |
bool CpuOffloadParams { get; } |
Gets whether to page the (sharded) parameter tensors to CPU RAM between training steps. |
bool EnableGradientCompression { get; } |
Gets whether to enable gradient compression to reduce communication costs. |
bool OverlapCommunication { get; } |
Gets whether backward gradient computation overlaps with asynchronous bucketed collective communication (AllReduceAsync / ReduceScatterAsync). |
int GradientBucketSizeBytes { get; } |
Gets the target capacity in bytes of each gradient communication bucket when overlapping gradient synchronization with backward propagation. |
int MinimumParameterGroupSize { get; } |
Gets the minimum parameter group size for sharding. |