GlooCommunicationBackend<T>

Models & Types · AiDotNet.DistributedTraining

Gloo-based communication backend for CPU-based collective operations.

Inheritance: CommunicationBackendBase<T> → GlooCommunicationBackend<T>

How It Works

Overview: Gloo is Facebook’s collective communications library optimized for both CPUs and GPUs. It provides efficient implementations of collective operations for CPU-based training or heterogeneous environments. Gloo is particularly well-suited for training on CPUs or mixed CPU/GPU clusters where NCCL may not be available or optimal.

Features:

  • CPU-optimized collective operations
  • Supports TCP, InfiniBand via ibverbs
  • Works on both CPUs and GPUs
  • Cross-platform (Linux, macOS, Windows)
  • Used by PyTorch’s distributed package

Use Cases:

  • CPU-based distributed training
  • Heterogeneous clusters (mixed CPU/GPU)
  • When NCCL is not available (non-NVIDIA hardware, macOS, etc.)
  • Development and testing on laptops/workstations
  • Production training on CPU clusters

Requirements:

  • Gloo library (C++)
  • .NET bindings for Gloo (custom P/Invoke or wrapper library)
  • Network connectivity between workers (TCP/IP or InfiniBand)

Architecture: This backend supports two modes of operation:

  1. Native Gloo Mode (Optional):

Requires GlooSharp package (separate NuGet) which provides .NET bindings for the native Gloo C++ library. Gloo offers optimized collective operations for CPU and GPU. To use: Install the GlooSharp package separately.

  1. Built-in TCP Mode (Default, Production-Ready):

Production-ready TCP-based implementation using industry-standard ring algorithms (ring-allreduce, ring-allgather, ring-reduce-scatter). Provides full multi-process functionality without external dependencies.

The TCP implementation features:

  • Automatic TCP connection initialization with retry logic and handshakes
  • Ring-based collective operations for optimal bandwidth utilization
  • Zero-LOH pooled buffers, direct span socket I/O, and SIMD reductions
  • Proper error handling, validation, and timeout mechanisms
  • Environment-based rendezvous (AIDOTNET_MASTER_ADDR, AIDOTNET_MASTER_PORT)
  • Support for arbitrary world sizes and fault-tolerant connection establishment

Environment Variables:

  • AIDOTNET_GLOO_TRANSPORT: Transport to use (“tcp” or “ib”/“infiniband”). Default: “tcp”.
  • AIDOTNET_GLOO_IB_DEVICE: InfiniBand device name (only when transport is “ib”).
  • AIDOTNET_GLOO_STORE_PATH: Filesystem path for Gloo rendezvous/store coordination.
  • AIDOTNET_MASTER_ADDR: IP address of rank 0 for TCP rendezvous.
  • AIDOTNET_MASTER_PORT: Base port number for TCP connections (each rank uses port + rank).

Recommendation: Use TCP mode for most scenarios. Add GlooSharp only if you need specialized hardware support (InfiniBand) or have specific Gloo optimizations.

Constructors

Constructor Summary
GlooCommunicationBackend(int rank, int worldSize) Creates a new Gloo communication backend using production-ready TCP implementation.

Properties

Property Summary
int Rank { get; }
int WorldSize { get; }

Methods

Method Summary
Vector<T> AllGather(Vector<T> sendData)
Vector<T> Broadcast(Vector<T> data, int root)
Vector<T> Receive(int sourceRank, int count, int tag)
Vector<T> ReduceScatter(Vector<T> data, ReductionOperation operation)
Vector<T> Scatter(Vector<T> sendData, int root)
void AllReduce(Vector<T> data, ReductionOperation operation)
void Barrier()
void Send(Vector<T> data, int destinationRank, int tag)