DistributedStrategy

Enums · AiDotNet.Enums

Defines the distributed training strategy to use.

For Beginners

These strategies determine how your model and data are split across multiple GPUs or machines. DDP (Distributed Data Parallel) is the most common and works for 90% of use cases.

Fields

Field Summary
DistributedStrategy DDP Distributed Data Parallel (DDP) - Most common strategy.
DistributedStrategy FSDP Fully Sharded Data Parallel (FSDP) - PyTorch style, full parameter sharding.
DistributedStrategy Hybrid Hybrid - 3D parallelism combining data + tensor + pipeline.
DistributedStrategy None No distributed training - Single device training (default).
DistributedStrategy PipelineParallel Pipeline Parallel - Model split into stages across GPUs.
DistributedStrategy TensorParallel Tensor Parallel - Individual layers split across GPUs (Megatron-LM style).
DistributedStrategy ZeRO1 ZeRO Stage 1 - Only optimizer states are sharded.
DistributedStrategy ZeRO2 ZeRO Stage 2 - Optimizer states + gradients are sharded.
DistributedStrategy ZeRO3 ZeRO Stage 3 - Full sharding (equivalent to FSDP).
int value__