NCCLCommunicationBackend<T>

Models & Types ยท AiDotNet.DistributedTraining

NVIDIA NCCL-based communication backend for GPU-to-GPU communication.

Inheritance: CommunicationBackendBase<T> โ†’ NCCLCommunicationBackend<T>

How It Works

Overview: NCCL (NVIDIA Collective Communications Library) is optimized for multi-GPU communication on NVIDIA GPUs. It provides highly optimized implementations of collective operations that take advantage of NVLink, PCIe, and network topology for maximum throughput.

Features:

  • Optimized for NVIDIA GPUs (NVLink, NVSwitch awareness)
  • Near-optimal bandwidth utilization
  • Supports multi-node multi-GPU configurations
  • Ring and tree algorithms for different collective operations
  • Essential for high-performance multi-GPU training

Architecture: This backend supports two modes of operation:

  1. Native NCCL Mode:

Uses NCCL library with actual GPU memory for collective operations. Requires CUDA toolkit and NCCL library. Provides near-optimal GPU bandwidth.

  1. CPU Fallback Mode:

When NCCL/CUDA not available, uses TCP-based ring algorithms similar to Gloo. Allows development and testing on systems without NVIDIA GPUs.

The implementation features:

  • Automatic NCCL detection and initialization
  • TCP-based unique ID distribution for multi-node setup
  • Environment-based rendezvous (AIDOTNET_MASTER_ADDR, AIDOTNET_MASTER_PORT)
  • Proper CUDA stream synchronization
  • Memory-efficient GPU operations

Requirements for GPU Mode:

  • NVIDIA GPUs (compute capability 3.0+)
  • CUDA toolkit 10.0+
  • NCCL library 2.0+

Constructors

Constructor Summary
NCCLCommunicationBackend(int rank, int worldSize, int deviceId) Creates a new NCCL communication backend.

Properties

Property Summary
int Rank { get; }
int WorldSize { get; }

Methods

Method Summary
Vector<T> AllGather(Vector<T> sendData)
Vector<T> Broadcast(Vector<T> data, int root)
Vector<T> Receive(int sourceRank, int count, int tag)
Vector<T> ReduceScatter(Vector<T> data, ReductionOperation operation)
Vector<T> Scatter(Vector<T> sendData, int root)
void AllReduce(Vector<T> data, ReductionOperation operation)
void Barrier()
void Send(Vector<T> data, int destinationRank, int tag)