IShardingConfiguration<T>

Interfaces · AiDotNet.DistributedTraining

Configuration for parameter sharding in distributed training.

Known implementations: ShardingConfiguration<T>

For Beginners

This configuration tells the sharding system how to divide up parameters and how to handle communication. Think of it as the “rules” for how the team collaborates.

Properties

Property Summary
ICommunicationBackend<T> CommunicationBackend { get; } Gets the communication backend to use for distributed operations.
T LearningRate { get; } Gets the learning rate for gradient application during training.
bool AutoSyncGradients { get; } Gets whether to automatically synchronize gradients after backward pass.
bool CpuOffloadGradients { get; } Gets whether to page gradient tensors to CPU RAM before the sharded reduction (all-reduce for DDP, reduce-scatter for FSDP/ZeRO).
bool CpuOffloadOptimizer { get; } Gets whether to keep optimizer state (Adam’s m/v moments, momentum buffers, etc.) on the CPU and run the optimizer step on the CPU too.
bool CpuOffloadParams { get; } Gets whether to page the (sharded) parameter tensors to CPU RAM between training steps.
bool EnableGradientCompression { get; } Gets whether to enable gradient compression to reduce communication costs.
bool OverlapCommunication { get; } Gets whether backward gradient computation overlaps with asynchronous bucketed collective communication (AllReduceAsync / ReduceScatterAsync).
int GradientBucketSizeBytes { get; } Gets the target capacity in bytes of each gradient communication bucket when overlapping gradient synchronization with backward propagation.
int MinimumParameterGroupSize { get; } Gets the minimum parameter group size for sharding.