SharedMemoryCommunicationBackend<T>
Models & Types · AiDotNet.DistributedTraining
Provides a high-throughput, low-latency inter-process shared-memory communication backend for multi-GPU training on a single machine (specifically designed for Windows where native NCCL is unavailable, with cross-platform POSIX shared-file memory mapping on Linux and macOS).
Inheritance: CommunicationBackendBase<T> → SharedMemoryCommunicationBackend<T>
Implements: IDisposable
How It Works
Architecture:
Processes on the same host map a common shared-memory region (MemoryMappedFile) partitioned into:
- Cache-line-aligned (64-byte) per-rank sense-reversing phase headers to avoid false sharing.
- Dedicated per-rank collective staging slots supporting single-pass zero-copy SIMD and pipelined
multi-chunk AllReduce, ReduceScatter, AllGather (both uniform and uneven shard sizes),
Broadcast, and Scatter.
- Dedicated per-pair point-to-point mailboxes for tagged
Send/Receive. - OS process-ID liveness monitoring so if any peer rank crashes or is killed mid-run, surviving
ranks fail fast with an IOException instead of hanging indefinitely.
Determinism:
Reductions combine peer contributions in strict canonical rank order (0, 1, ..., WorldSize - 1) on every
rank, guaranteeing bit-identical floating-point results across all participating processes.
Constructors
| Constructor | Summary |
|---|---|
SharedMemoryCommunicationBackend() |
Initializes a new SharedMemoryCommunicationBackend configured from environment variables (AIDOTNET_RANK or RANK, AIDOTNET_WORLD_SIZE or WORLD_SIZE, and AIDOTNET_SHM_NAME). |
SharedMemoryCommunicationBackend(int rank, int worldSize, string sessionName, long slotCapacityBytes, int timeoutMilliseconds) |
Initializes a new SharedMemoryCommunicationBackend for a specific rank and world size. |
Properties
| Property | Summary |
|---|---|
int Rank { get; } |
|
int WorldSize { get; } |
|
string SessionName { get; } |
Gets the sanitized shared-memory session identifier used by this backend. |
Methods
| Method | Summary |
|---|---|
Vector<T> AllGather(Vector<T> sendData) |
|
Vector<T> Broadcast(Vector<T> data, int root) |
|
Vector<T> Receive(int sourceRank, int count, int tag) |
|
Vector<T> ReduceScatter(Vector<T> data, ReductionOperation operation) |
|
Vector<T> Scatter(Vector<T> sendData, int root) |
|
void AllReduce(Vector<T> data, ReductionOperation operation) |
|
void Barrier() |
|
void Dispose() |
|
void Send(Vector<T> data, int destinationRank, int tag) |