TensorParallelInference

Helpers & Utilities · AiDotNet.DistributedTraining

Runs a tensor-parallel forward pass for inference (serving), executing all tensor-parallel ranks so their collective operations (column-parallel gather, row-parallel all-reduce) rendezvous and combine into the same fully-reduced output every rank sees.

For Beginners

A very large model may not fit (or run fast enough) on a single GPU. Tensor parallelism splits each layer’s weights across several GPUs so they share the work on every token. This helper runs all those splits together and stitches the answer back into exactly what the whole model would have produced.

How It Works

Tensor parallelism shards each linear layer’s weights across ranks (Megatron style): a ColumnParallelLinear owns a slice of the output columns, and a following RowParallelLinear owns the matching input rows and all-reduces the partial sums. This runner drives every rank’s forward and returns the combined result — the model output is identical to running the un-sharded model, but the per-rank compute and weight memory are divided across the ranks.

ForwardInProcess simulates the ranks in ONE process (one task per rank over an InMemoryCommunicationBackend), which is how single-node multi-device / test scenarios run and how the sharded-equals-unsharded equivalence is verified. A real multi-node deployment instead runs one process per rank against a networked backend (MPI/NCCL/Gloo) and calls the same layer forwards.

Methods

Method Summary
Tensor<T> ForwardInProcess(int worldSize, Tensor<T> input, Func<ICommunicationBackend<T>, IReadOnlyList<ILayer<T>>> buildRank) Runs a tensor-parallel forward across worldSize ranks in-process (one task per rank over a shared InMemoryCommunicationBackend) and returns the combined output.