ICommunicationBackend<T>

Interfaces · AiDotNet.DistributedTraining

Defines the contract for distributed communication backends.

Known implementations: IAsyncCommunicationBackend<T>

For Beginners

This interface defines how different processes (or GPUs) communicate with each other during distributed training. Think of it as a “walkie-talkie” system where multiple processes can send data to each other, synchronize, and perform collective operations.

How It Works

This abstraction allows different implementations (in-memory, MPI.NET, NCCL, etc.) to provide collective communication operations for distributed training.

Properties

Property Summary
bool IsInitialized { get; } Gets whether this backend is initialized and ready for use.
int Rank { get; } Gets the rank (ID) of the current process in the distributed group.
int WorldSize { get; } Gets the total number of processes in the distributed group.

Methods

Method Summary
Vector<T> AllGather(Vector<T> sendData) AllGather operation - gathers data from all processes and concatenates it.
Vector<T> Broadcast(Vector<T> data, int root) Broadcast operation - sends data from one process (root) to all other processes.
Vector<T> Receive(int sourceRank, int count, int tag) Receive operation - receives data from a specific source process.
Vector<T> ReduceScatter(Vector<T> data, ReductionOperation operation) ReduceScatter operation - reduces data and scatters the result.
Vector<T> Scatter(Vector<T> sendData, int root) Scatter operation - distributes different chunks of data from root to each process.
void AllReduce(Vector<T> data, ReductionOperation operation) AllReduce operation - combines data from all processes using the specified operation and distributes the result back to all processes.
void Barrier() Synchronization barrier - blocks until all processes reach this point.
void Initialize() Initializes the communication backend.
void Send(Vector<T> data, int destinationRank, int tag) Send operation - sends data from this process to a specific destination process.
void Shutdown() Shuts down the communication backend and releases resources.