DistributedStrategy
Enums · AiDotNet.Enums
Defines the distributed training strategy to use.
For Beginners
These strategies determine how your model and data are split across multiple GPUs or machines. DDP (Distributed Data Parallel) is the most common and works for 90% of use cases.
Fields
| Field | Summary |
|---|---|
DistributedStrategy DDP |
Distributed Data Parallel (DDP) - Most common strategy. |
DistributedStrategy FSDP |
Fully Sharded Data Parallel (FSDP) - PyTorch style, full parameter sharding. |
DistributedStrategy Hybrid |
Hybrid - 3D parallelism combining data + tensor + pipeline. |
DistributedStrategy None |
No distributed training - Single device training (default). |
DistributedStrategy PipelineParallel |
Pipeline Parallel - Model split into stages across GPUs. |
DistributedStrategy TensorParallel |
Tensor Parallel - Individual layers split across GPUs (Megatron-LM style). |
DistributedStrategy ZeRO1 |
ZeRO Stage 1 - Only optimizer states are sharded. |
DistributedStrategy ZeRO2 |
ZeRO Stage 2 - Optimizer states + gradients are sharded. |
DistributedStrategy ZeRO3 |
ZeRO Stage 3 - Full sharding (equivalent to FSDP). |
int value__ |