GlooCommunicationBackend<T>
Models & Types · AiDotNet.DistributedTraining
Gloo-based communication backend for CPU-based collective operations.
Inheritance: CommunicationBackendBase<T> → GlooCommunicationBackend<T>
How It Works
Overview: Gloo is Facebook’s collective communications library optimized for both CPUs and GPUs. It provides efficient implementations of collective operations for CPU-based training or heterogeneous environments. Gloo is particularly well-suited for training on CPUs or mixed CPU/GPU clusters where NCCL may not be available or optimal.
Features:
- CPU-optimized collective operations
- Supports TCP, InfiniBand via ibverbs
- Works on both CPUs and GPUs
- Cross-platform (Linux, macOS, Windows)
- Used by PyTorch’s distributed package
Use Cases:
- CPU-based distributed training
- Heterogeneous clusters (mixed CPU/GPU)
- When NCCL is not available (non-NVIDIA hardware, macOS, etc.)
- Development and testing on laptops/workstations
- Production training on CPU clusters
Requirements:
- Gloo library (C++)
- .NET bindings for Gloo (custom P/Invoke or wrapper library)
- Network connectivity between workers (TCP/IP or InfiniBand)
Architecture: This backend supports two modes of operation:
- Native Gloo Mode (Optional):
Requires GlooSharp package (separate NuGet) which provides .NET bindings for the native Gloo C++ library. Gloo offers optimized collective operations for CPU and GPU. To use: Install the GlooSharp package separately.
- Built-in TCP Mode (Default, Production-Ready):
Production-ready TCP-based implementation using industry-standard ring algorithms (ring-allreduce, ring-allgather, ring-reduce-scatter). Provides full multi-process functionality without external dependencies.
The TCP implementation features:
- Automatic TCP connection initialization with retry logic and handshakes
- Ring-based collective operations for optimal bandwidth utilization
- Zero-LOH pooled buffers, direct span socket I/O, and SIMD reductions
- Proper error handling, validation, and timeout mechanisms
- Environment-based rendezvous (AIDOTNET_MASTER_ADDR, AIDOTNET_MASTER_PORT)
- Support for arbitrary world sizes and fault-tolerant connection establishment
Environment Variables:
- AIDOTNET_GLOO_TRANSPORT: Transport to use (“tcp” or “ib”/“infiniband”). Default: “tcp”.
- AIDOTNET_GLOO_IB_DEVICE: InfiniBand device name (only when transport is “ib”).
- AIDOTNET_GLOO_STORE_PATH: Filesystem path for Gloo rendezvous/store coordination.
- AIDOTNET_MASTER_ADDR: IP address of rank 0 for TCP rendezvous.
- AIDOTNET_MASTER_PORT: Base port number for TCP connections (each rank uses port + rank).
Recommendation: Use TCP mode for most scenarios. Add GlooSharp only if you need specialized hardware support (InfiniBand) or have specific Gloo optimizations.
Constructors
| Constructor | Summary |
|---|---|
GlooCommunicationBackend(int rank, int worldSize) |
Creates a new Gloo communication backend using production-ready TCP implementation. |
Properties
| Property | Summary |
|---|---|
int Rank { get; } |
|
int WorldSize { get; } |
Methods
| Method | Summary |
|---|---|
Vector<T> AllGather(Vector<T> sendData) |
|
Vector<T> Broadcast(Vector<T> data, int root) |
|
Vector<T> Receive(int sourceRank, int count, int tag) |
|
Vector<T> ReduceScatter(Vector<T> data, ReductionOperation operation) |
|
Vector<T> Scatter(Vector<T> sendData, int root) |
|
void AllReduce(Vector<T> data, ReductionOperation operation) |
|
void Barrier() |
|
void Send(Vector<T> data, int destinationRank, int tag) |