ShardingConfiguration<T>
Options & Configuration · AiDotNet.DistributedTraining
Default implementation of sharding configuration for distributed training.
Implements: IShardingConfiguration<T>
For Beginners
This class holds all the settings that control how distributed training works. You can create an instance with default settings or customize it for your needs.
How It Works
Example:
Constructors
| Constructor | Summary |
|---|---|
ShardingConfiguration(ICommunicationBackend<T> communicationBackend, double learningRate) |
Creates a new sharding configuration with the specified communication backend. |
Properties
| Property | Summary |
|---|---|
ICommunicationBackend<T> CommunicationBackend { get; } |
|
T LearningRate { get; set; } |
|
bool AutoSyncGradients { get; set; } |
|
bool CpuOffloadGradients { get; set; } |
|
bool CpuOffloadOptimizer { get; set; } |
|
bool CpuOffloadParams { get; set; } |
|
bool EnableGradientCompression { get; set; } |
|
bool OverlapCommunication { get; set; } |
|
int GradientBucketSizeBytes { get; set; } |
|
int MinimumParameterGroupSize { get; set; } |
Methods
| Method | Summary |
|---|---|
ShardingConfiguration<T> CreateDefault(ICommunicationBackend<T> communicationBackend) |
Creates a new sharding configuration with default settings and the specified backend. |
ShardingConfiguration<T> CreateForHighBandwidth(ICommunicationBackend<T> communicationBackend) |
Creates a configuration optimized for high-bandwidth networks (like NVLink between GPUs). |
ShardingConfiguration<T> CreateForLowBandwidth(ICommunicationBackend<T> communicationBackend) |
Creates a configuration optimized for low-bandwidth networks (like machines connected over ethernet). |
ShardingConfiguration<T> CreateForZeROOffload(ICommunicationBackend<T> communicationBackend) |
Creates a configuration equivalent to DeepSpeed ZeRO-Offload Stage-1: keeps optimizer state (Adam m/v, momentum buffers) on the CPU and runs the optimizer step on the CPU. |
ShardingConfiguration<T> CreateForZeROOffloadFull(ICommunicationBackend<T> communicationBackend) |
Enables all three CPU-offload contracts at once: optimizer state (real ZeRO-1/2 state partitioning + CPU-resident Adam update), gradients (CPU-resident during the reduce), and parameters. |