NCCLCommunicationBackend<T>
Models & Types ยท AiDotNet.DistributedTraining
NVIDIA NCCL-based communication backend for GPU-to-GPU communication.
Inheritance: CommunicationBackendBase<T> โ NCCLCommunicationBackend<T>
How It Works
Overview: NCCL (NVIDIA Collective Communications Library) is optimized for multi-GPU communication on NVIDIA GPUs. It provides highly optimized implementations of collective operations that take advantage of NVLink, PCIe, and network topology for maximum throughput.
Features:
- Optimized for NVIDIA GPUs (NVLink, NVSwitch awareness)
- Near-optimal bandwidth utilization
- Supports multi-node multi-GPU configurations
- Ring and tree algorithms for different collective operations
- Essential for high-performance multi-GPU training
Architecture: This backend supports two modes of operation:
- Native NCCL Mode:
Uses NCCL library with actual GPU memory for collective operations. Requires CUDA toolkit and NCCL library. Provides near-optimal GPU bandwidth.
- CPU Fallback Mode:
When NCCL/CUDA not available, uses TCP-based ring algorithms similar to Gloo. Allows development and testing on systems without NVIDIA GPUs.
The implementation features:
- Automatic NCCL detection and initialization
- TCP-based unique ID distribution for multi-node setup
- Environment-based rendezvous (AIDOTNET_MASTER_ADDR, AIDOTNET_MASTER_PORT)
- Proper CUDA stream synchronization
- Memory-efficient GPU operations
Requirements for GPU Mode:
- NVIDIA GPUs (compute capability 3.0+)
- CUDA toolkit 10.0+
- NCCL library 2.0+
Constructors
| Constructor | Summary |
|---|---|
NCCLCommunicationBackend(int rank, int worldSize, int deviceId) |
Creates a new NCCL communication backend. |
Properties
| Property | Summary |
|---|---|
int Rank { get; } |
|
int WorldSize { get; } |
Methods
| Method | Summary |
|---|---|
Vector<T> AllGather(Vector<T> sendData) |
|
Vector<T> Broadcast(Vector<T> data, int root) |
|
Vector<T> Receive(int sourceRank, int count, int tag) |
|
Vector<T> ReduceScatter(Vector<T> data, ReductionOperation operation) |
|
Vector<T> Scatter(Vector<T> sendData, int root) |
|
void AllReduce(Vector<T> data, ReductionOperation operation) |
|
void Barrier() |
|
void Send(Vector<T> data, int destinationRank, int tag) |