Distributed Training

58 public types in this namespace, organized by kind.

Models & Types (36)

Type Summary
AsyncSGDOptimizer<T, TInput, TOutput> Implements Asynchronous SGD optimizer - allows asynchronous parameter updates without strict barriers.
DDPModel<T, TInput, TOutput> Implements DDP (Distributed Data Parallel) model wrapper for distributed training.
DDPOptimizer<T, TInput, TOutput> Implements true DDP (Distributed Data Parallel) optimizer - industry-standard gradient averaging.
ElasticOptimizer<T, TInput, TOutput> Implements Elastic optimizer - supports dynamic worker addition/removal during training.
FSDPModel<T, TInput, TOutput> Implements FSDP (Fully Sharded Data Parallel) model wrapper that shards parameters across multiple processes.
FSDPOptimizer<T, TInput, TOutput> Implements FSDP (Fully Sharded Data Parallel) optimizer wrapper that coordinates optimization across multiple processes.
GPipeSchedule<T> Implements the GPipe scheduling strategy: all forward passes first, then all backward passes.
GlooCommunicationBackend<T> Gloo-based communication backend for CPU-based collective operations.
GradientCompressionOptimizer<T, TInput, TOutput> Implements gradient compression optimizer - wraps any optimizer to add gradient compression.
HybridShardedModel<T, TInput, TOutput> Implements 3D Parallelism (Hybrid Sharded) model - combines data, tensor, and pipeline parallelism.
HybridShardedOptimizer<T, TInput, TOutput> Implements 3D Parallelism optimizer - coordinates across data, tensor, and pipeline dimensions.
InMemoryCommunicationBackend<T> Provides an in-memory implementation of distributed communication for testing and single-machine scenarios.
Interleaved1F1BSchedule<T> Implements the Interleaved 1F1B pipeline schedule with multiple virtual stages per rank.
LoadBalancedPartitionStrategy<T> Partitions model parameters across pipeline stages using estimated computational cost per layer.
LocalSGDOptimizer<T, TInput, TOutput> Implements Local SGD distributed training optimizer - parameter averaging after local optimization.
LoopedBFSSchedule<T> Implements the Looped BFS (Breadth-First Schedule) pipeline schedule with multiple virtual stages per rank.
MPICommunicationBackend<T> MPI.NET-based communication backend for production distributed training.
NCCLCommunicationBackend<T> NVIDIA NCCL-based communication backend for GPU-to-GPU communication.
OneForwardOneBackwardSchedule<T> Implements the 1F1B (One-Forward-One-Backward) pipeline schedule.
ParameterAnalyzer<T> Analyzes model parameters and creates optimized groupings for distributed communication.
ParameterGroup<T> Represents a group of parameters that should be communicated together.
PipelineParallelModel<T, TInput, TOutput> Implements Pipeline Parallel model wrapper - splits model into stages across ranks.
PipelineParallelOptimizer<T, TInput, TOutput> Implements Pipeline Parallel optimizer - coordinates optimization across pipeline stages.
SharedMemoryCommunicationBackend<T> Provides a high-throughput, low-latency inter-process shared-memory communication backend for multi-GPU training on a single machine (specifically designed for Windows where native NCCL is unavailable, with cross-platform POSIX shared-file…
TensorParallelModel<T, TInput, TOutput> Implements Tensor Parallel model wrapper - splits individual layers across ranks (Megatron-LM style).
TensorParallelOptimizer<T, TInput, TOutput> Implements Tensor Parallel optimizer - coordinates updates for tensor-parallel layers.
UniformPartitionStrategy<T> Divides model parameters evenly across pipeline stages.
ZeRO1Model<T, TInput, TOutput> Implements ZeRO Stage 1 model wrapper - shards optimizer states only.
ZeRO1Optimizer<T, TInput, TOutput> Implements ZeRO Stage 1 optimizer - shards optimizer states only.
ZeRO2Model<T, TInput, TOutput> Implements ZeRO Stage 2 model wrapper - shards optimizer states and gradients.
ZeRO2Optimizer<T, TInput, TOutput> Implements ZeRO Stage 2 optimizer - shards gradients and optimizer states across ranks.
ZeRO3Model<T, TInput, TOutput> Implements ZeRO Stage 3 model wrapper - full sharding of parameters, gradients, and optimizer states.
ZeRO3Optimizer<T, TInput, TOutput> Implements ZeRO Stage 3 optimizer - full sharding equivalent to FSDP.
ZeroBubbleH1Schedule<T> Implements the Zero Bubble H1 (ZB-H1) pipeline schedule.
ZeroBubbleH2Schedule<T> Implements the Zero Bubble H2 (ZB-H2) pipeline schedule.
ZeroBubbleVSchedule<T> Implements the Zero Bubble V (ZB-V) pipeline schedule with 2 virtual stages per rank.

Layers (5)

Type Summary
ColumnParallelLinear<T> Megatron-LM column-parallel linear layer (Shoeybi et al.
RowParallelLinear<T> Megatron-LM row-parallel linear layer (Shoeybi et al.
Stage3ShardedLinear<T> A linear layer with true ZeRO Stage-3 / FSDP parameter RESIDENCY (Rajbhandari et al.
TensorParallelAttention<T> Megatron-style tensor-parallel multi-head self-attention for inference/serving: the Q/K/V projections are ColumnParallelLinear (each rank owns a contiguous group of attention heads), each rank runs scaled-dot-product attention over ONLY i…
TensorParallelTransformerBlock<T> A complete Megatron-style tensor-parallel transformer decoder block for inference/serving (pre-LayerNorm, GPT-style): h + Attn(LN1(h)) then h + MLP(LN2(h)), where the attention is a TensorParallelAttention and the MLP is the column-pa…

Base Classes (3)

Type Summary
CommunicationBackendBase<T> Provides base implementation for distributed communication backends.
ShardedModelBase<T, TInput, TOutput> Provides base implementation for distributed models with parameter sharding.
ShardedOptimizerBase<T, TInput, TOutput> Provides base implementation for distributed optimizers with parameter sharding.

Interfaces (7)

Type Summary
IAsyncCollectiveHandle Represents an in-flight asynchronous collective communication operation.
IAsyncCollectiveResultHandle<T> Represents an in-flight asynchronous collective communication operation that produces a new vector result (such as ReduceScatter or AllGather).
IAsyncCommunicationBackend<T> Extends ICommunicationBackend with non-blocking collective operations and raw transport timing diagnostics so gradient bucketing can overlap communication with backward computation.
ICommunicationBackend<T> Defines the contract for distributed communication backends.
IShardedModel<T, TInput, TOutput> Defines the contract for models that support distributed training with parameter sharding.
IShardedOptimizer<T, TInput, TOutput> Defines the contract for optimizers that support distributed training with parameter sharding.
IShardingConfiguration<T> Configuration for parameter sharding in distributed training.

Enums (2)

Type Summary
RecomputeStrategy Strategy for recomputing activations during the backward pass.
ReductionOperation Defines the supported reduction operations for collective communication.

Options & Configuration (2)

Type Summary
ActivationCheckpointConfig Configuration for activation checkpointing in pipeline parallel training.
ShardingConfiguration<T> Default implementation of sharding configuration for distributed training.

Helpers & Utilities (3)

Type Summary
CommunicationManager Central manager for distributed communication operations.
DistributedExtensions Provides extension methods for easily enabling distributed training on models and optimizers.
TensorParallelInference Runs a tensor-parallel forward pass for inference (serving), executing all tensor-parallel ranks so their collective operations (column-parallel gather, row-parallel all-reduce) rendezvous and combine into the same fully-reduced output ever…