AsyncSGDOptimizer<T, TInput, TOutput> |
Implements Asynchronous SGD optimizer - allows asynchronous parameter updates without strict barriers. |
DDPModel<T, TInput, TOutput> |
Implements DDP (Distributed Data Parallel) model wrapper for distributed training. |
DDPOptimizer<T, TInput, TOutput> |
Implements true DDP (Distributed Data Parallel) optimizer - industry-standard gradient averaging. |
ElasticOptimizer<T, TInput, TOutput> |
Implements Elastic optimizer - supports dynamic worker addition/removal during training. |
FSDPModel<T, TInput, TOutput> |
Implements FSDP (Fully Sharded Data Parallel) model wrapper that shards parameters across multiple processes. |
FSDPOptimizer<T, TInput, TOutput> |
Implements FSDP (Fully Sharded Data Parallel) optimizer wrapper that coordinates optimization across multiple processes. |
GPipeSchedule<T> |
Implements the GPipe scheduling strategy: all forward passes first, then all backward passes. |
GlooCommunicationBackend<T> |
Gloo-based communication backend for CPU-based collective operations. |
GradientCompressionOptimizer<T, TInput, TOutput> |
Implements gradient compression optimizer - wraps any optimizer to add gradient compression. |
HybridShardedModel<T, TInput, TOutput> |
Implements 3D Parallelism (Hybrid Sharded) model - combines data, tensor, and pipeline parallelism. |
HybridShardedOptimizer<T, TInput, TOutput> |
Implements 3D Parallelism optimizer - coordinates across data, tensor, and pipeline dimensions. |
InMemoryCommunicationBackend<T> |
Provides an in-memory implementation of distributed communication for testing and single-machine scenarios. |
Interleaved1F1BSchedule<T> |
Implements the Interleaved 1F1B pipeline schedule with multiple virtual stages per rank. |
LoadBalancedPartitionStrategy<T> |
Partitions model parameters across pipeline stages using estimated computational cost per layer. |
LocalSGDOptimizer<T, TInput, TOutput> |
Implements Local SGD distributed training optimizer - parameter averaging after local optimization. |
LoopedBFSSchedule<T> |
Implements the Looped BFS (Breadth-First Schedule) pipeline schedule with multiple virtual stages per rank. |
MPICommunicationBackend<T> |
MPI.NET-based communication backend for production distributed training. |
NCCLCommunicationBackend<T> |
NVIDIA NCCL-based communication backend for GPU-to-GPU communication. |
OneForwardOneBackwardSchedule<T> |
Implements the 1F1B (One-Forward-One-Backward) pipeline schedule. |
ParameterAnalyzer<T> |
Analyzes model parameters and creates optimized groupings for distributed communication. |
ParameterGroup<T> |
Represents a group of parameters that should be communicated together. |
PipelineParallelModel<T, TInput, TOutput> |
Implements Pipeline Parallel model wrapper - splits model into stages across ranks. |
PipelineParallelOptimizer<T, TInput, TOutput> |
Implements Pipeline Parallel optimizer - coordinates optimization across pipeline stages. |
SharedMemoryCommunicationBackend<T> |
Provides a high-throughput, low-latency inter-process shared-memory communication backend for multi-GPU training on a single machine (specifically designed for Windows where native NCCL is unavailable, with cross-platform POSIX shared-file… |
TensorParallelModel<T, TInput, TOutput> |
Implements Tensor Parallel model wrapper - splits individual layers across ranks (Megatron-LM style). |
TensorParallelOptimizer<T, TInput, TOutput> |
Implements Tensor Parallel optimizer - coordinates updates for tensor-parallel layers. |
UniformPartitionStrategy<T> |
Divides model parameters evenly across pipeline stages. |
ZeRO1Model<T, TInput, TOutput> |
Implements ZeRO Stage 1 model wrapper - shards optimizer states only. |
ZeRO1Optimizer<T, TInput, TOutput> |
Implements ZeRO Stage 1 optimizer - shards optimizer states only. |
ZeRO2Model<T, TInput, TOutput> |
Implements ZeRO Stage 2 model wrapper - shards optimizer states and gradients. |
ZeRO2Optimizer<T, TInput, TOutput> |
Implements ZeRO Stage 2 optimizer - shards gradients and optimizer states across ranks. |
ZeRO3Model<T, TInput, TOutput> |
Implements ZeRO Stage 3 model wrapper - full sharding of parameters, gradients, and optimizer states. |
ZeRO3Optimizer<T, TInput, TOutput> |
Implements ZeRO Stage 3 optimizer - full sharding equivalent to FSDP. |
ZeroBubbleH1Schedule<T> |
Implements the Zero Bubble H1 (ZB-H1) pipeline schedule. |
ZeroBubbleH2Schedule<T> |
Implements the Zero Bubble H2 (ZB-H2) pipeline schedule. |
ZeroBubbleVSchedule<T> |
Implements the Zero Bubble V (ZB-V) pipeline schedule with 2 virtual stages per rank. |