SharedMemoryCommunicationBackend<T>

Models & Types · AiDotNet.DistributedTraining

Provides a high-throughput, low-latency inter-process shared-memory communication backend for multi-GPU training on a single machine (specifically designed for Windows where native NCCL is unavailable, with cross-platform POSIX shared-file memory mapping on Linux and macOS).

Inheritance: CommunicationBackendBase<T> → SharedMemoryCommunicationBackend<T>

Implements: IDisposable

How It Works

Architecture: Processes on the same host map a common shared-memory region (MemoryMappedFile) partitioned into:

  • Cache-line-aligned (64-byte) per-rank sense-reversing phase headers to avoid false sharing.
  • Dedicated per-rank collective staging slots supporting single-pass zero-copy SIMD and pipelined

multi-chunk AllReduce, ReduceScatter, AllGather (both uniform and uneven shard sizes), Broadcast, and Scatter.

  • Dedicated per-pair point-to-point mailboxes for tagged Send/Receive.
  • OS process-ID liveness monitoring so if any peer rank crashes or is killed mid-run, surviving

ranks fail fast with an IOException instead of hanging indefinitely.

Determinism: Reductions combine peer contributions in strict canonical rank order (0, 1, ..., WorldSize - 1) on every rank, guaranteeing bit-identical floating-point results across all participating processes.

Constructors

Constructor Summary
SharedMemoryCommunicationBackend() Initializes a new SharedMemoryCommunicationBackend configured from environment variables (AIDOTNET_RANK or RANK, AIDOTNET_WORLD_SIZE or WORLD_SIZE, and AIDOTNET_SHM_NAME).
SharedMemoryCommunicationBackend(int rank, int worldSize, string sessionName, long slotCapacityBytes, int timeoutMilliseconds) Initializes a new SharedMemoryCommunicationBackend for a specific rank and world size.

Properties

Property Summary
int Rank { get; }
int WorldSize { get; }
string SessionName { get; } Gets the sanitized shared-memory session identifier used by this backend.

Methods

Method Summary
Vector<T> AllGather(Vector<T> sendData)
Vector<T> Broadcast(Vector<T> data, int root)
Vector<T> Receive(int sourceRank, int count, int tag)
Vector<T> ReduceScatter(Vector<T> data, ReductionOperation operation)
Vector<T> Scatter(Vector<T> sendData, int root)
void AllReduce(Vector<T> data, ReductionOperation operation)
void Barrier()
void Dispose()
void Send(Vector<T> data, int destinationRank, int tag)