ICommunicationBackend<T>
Interfaces · AiDotNet.DistributedTraining
Defines the contract for distributed communication backends.
Known implementations: IAsyncCommunicationBackend<T>
For Beginners
This interface defines how different processes (or GPUs) communicate with each other during distributed training. Think of it as a “walkie-talkie” system where multiple processes can send data to each other, synchronize, and perform collective operations.
How It Works
This abstraction allows different implementations (in-memory, MPI.NET, NCCL, etc.) to provide collective communication operations for distributed training.
Properties
| Property | Summary |
|---|---|
bool IsInitialized { get; } |
Gets whether this backend is initialized and ready for use. |
int Rank { get; } |
Gets the rank (ID) of the current process in the distributed group. |
int WorldSize { get; } |
Gets the total number of processes in the distributed group. |
Methods
| Method | Summary |
|---|---|
Vector<T> AllGather(Vector<T> sendData) |
AllGather operation - gathers data from all processes and concatenates it. |
Vector<T> Broadcast(Vector<T> data, int root) |
Broadcast operation - sends data from one process (root) to all other processes. |
Vector<T> Receive(int sourceRank, int count, int tag) |
Receive operation - receives data from a specific source process. |
Vector<T> ReduceScatter(Vector<T> data, ReductionOperation operation) |
ReduceScatter operation - reduces data and scatters the result. |
Vector<T> Scatter(Vector<T> sendData, int root) |
Scatter operation - distributes different chunks of data from root to each process. |
void AllReduce(Vector<T> data, ReductionOperation operation) |
AllReduce operation - combines data from all processes using the specified operation and distributes the result back to all processes. |
void Barrier() |
Synchronization barrier - blocks until all processes reach this point. |
void Initialize() |
Initializes the communication backend. |
void Send(Vector<T> data, int destinationRank, int tag) |
Send operation - sends data from this process to a specific destination process. |
void Shutdown() |
Shuts down the communication backend and releases resources. |