CommunicationManager

Helpers & Utilities · AiDotNet.DistributedTraining

Central manager for distributed communication operations.

For Beginners

This is your main entry point for distributed training communication. It’s a “wrapper” that makes it easy to communicate between different processes/GPUs without worrying about the underlying implementation details.

How It Works

⚠️ WARNING - Static Mutable State: This class uses static mutable state for managing communication backends. This design choice has important implications for concurrent usage:

  • Only ONE backend can be active per process
  • Unit tests using this class CANNOT run in parallel
  • Multiple training sessions in the same process share the same backend

See detailed thread-safety notes below for proper usage patterns.

Provides a static API for collective communication in distributed training scenarios.

Example usage:

IMPORTANT - Thread Safety and Testing Limitations: This class uses STATIC MUTABLE STATE which has the following implications:

  1. SINGLE GLOBAL INSTANCE: Only ONE backend can be active per process at a time.

Multiple training sessions in the same process will share the same backend instance.

  1. PARALLEL TEST EXECUTION: Tests that use this class CANNOT run in parallel.

Use [Collection] attributes in xUnit or similar mechanisms to enforce sequential execution.

  1. TEST ISOLATION: Always call Shutdown() in test cleanup to reset state.

For better isolation in tests, use InMemoryCommunicationBackend with unique environment IDs and inject the backend directly instead of using this static manager.

  1. CONCURRENT INITIALIZATION: Attempting to Initialize() from multiple threads concurrently

is protected by locks, but may result in exceptions if already initialized.

Recommended Test Pattern:

Properties

Property Summary
bool IsInitialized { get; } Gets whether the communication manager has been initialized.

Methods

Method Summary
Vector<T> AllGather(Vector<T> sendData) Gathers data from all processes and returns the concatenated result.
Vector<T> Broadcast(Vector<T> data, int root) Broadcasts data from one process (root) to all other processes.
Vector<T> ReduceScatter(Vector<T> data, ReductionOperation operation) Performs a reduce-scatter operation - combines data and distributes chunks.
Vector<T> Scatter(Vector<T> sendData, int root) Scatters different chunks of data from root to each process.
int GetRank() Gets the rank (ID) of the current process.
int GetWorldSize() Gets the total number of processes in the distributed group.
void AllReduce(Vector<T> data, ReductionOperation operation) Performs an AllReduce operation - combines data from all processes and distributes the result to all processes.
void Barrier() Blocks until all processes reach this synchronization point.
void Initialize(ICommunicationBackend<T> backend) Initializes the communication manager with the specified backend.
void Shutdown() Shuts down the communication manager and releases all resources.